Nov 01 21:43:43.770938 ip-10-0-135-146 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 21:43:44.160289 ip-10-0-135-146 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:43:44.160289 ip-10-0-135-146 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 21:43:44.160289 ip-10-0-135-146 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:43:44.160289 ip-10-0-135-146 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:43:44.164384 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164312    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.164384 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164385    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.164384 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164389    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164392    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164396    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164399    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164402    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164404    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164407    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164410    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164412    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164415    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164418    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164420    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164423    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164431    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164434    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164437    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164439    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164442    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164444    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164447    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.164479 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164449    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164451    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164454    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164457    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164459    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164462    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164464    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164467    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164470    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164472    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164475    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164477    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164480    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164483    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164486    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164489    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164491    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164495    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164498    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.164968 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164500    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164503    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164505    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164508    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164510    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164513    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164515    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164518    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164520    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164523    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164526    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164528    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164530    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164533    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164537    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164541    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164544    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164547    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164549    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.165878 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164552    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164555    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164557    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164560    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164562    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164564    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164568    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164571    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164574    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164576    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164579    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164582    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164585    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164588    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164590    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164593    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164595    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164597    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164600    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164603    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.166338 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164605    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164608    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164611    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164613    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164616    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164618    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164621    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164623    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164627    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164629    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164632    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164635    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164638    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164641    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164643    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164646    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164648    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164651    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164653    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.166848 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164656    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164673    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164678    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164681    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164683    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164686    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.164689    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165470    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165477    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165480    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165482    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165485    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165488    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165491    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165494    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165496    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165499    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165502    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165506    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.167299 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165511    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165514    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165518    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165521    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165524    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165528    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165531    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165534    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165537    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165539    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165542    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165544    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165547    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165550    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165552    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165555    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165558    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165560    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165563    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165567    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.167810 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165576    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165579    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165582    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165584    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165587    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165589    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165592    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165600    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165603    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165605    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165608    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165611    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165613    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165616    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165619    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165621    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165624    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165627    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165629    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.168288 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165631    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165634    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165636    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165639    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165641    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165644    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165646    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165649    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165651    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165654    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165657    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165675    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165679    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165684    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165693    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165696    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165699    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165702    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165704    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.168752 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165707    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165710    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165712    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165715    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165718    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165720    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165723    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165726    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165728    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165731    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165733    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165736    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165739    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165741    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165744    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165746    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165749    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165751    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165754    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165756    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.169206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165759    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165761    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165764    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165766    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165772    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165776    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165778    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165781    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165789    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165792    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165794    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165797    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165799    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165803    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165805    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.165808    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166386    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166392    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166394    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.169716 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166397    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166399    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166402    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166405    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166407    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166410    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166412    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166415    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166418    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166420    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166423    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166425    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166428    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166431    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166433    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166436    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166438    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166443    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166447    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.170169 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166450    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166452    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166461    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166463    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166466    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166469    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166471    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166474    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166476    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166479    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166481    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166484    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166487    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166489    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166492    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166494    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166497    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166499    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166503    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166506    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.170611 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166509    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166512    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166514    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166517    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166520    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166523    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166525    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166527    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166530    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166532    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166536    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166540    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166543    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166545    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166548    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166557    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166560    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166562    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166565    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166567    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.171120 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166570    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166572    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166575    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166577    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166580    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166582    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166587    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166589    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166592    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166594    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166597    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166600    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166603    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166605    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166608    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166610    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166613    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166615    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166618    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166621    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.171623 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166623    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166626    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166630    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166634    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166636    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166639    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166641    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166644    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166647    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166649    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166652    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166654    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166670    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166673    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166676    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166678    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166681    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166683    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166686    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166688    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.172105 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166691    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166693    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166696    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.166700    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167595    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167604    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167613    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167618    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167622    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167626    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167630    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167634    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167637    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167641    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167646    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167650    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167653    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167656    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167672    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167675    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167678    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167682    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167687    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 01 21:43:44.172608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167690    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167693    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167696    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167700    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167704    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167707    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167710    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167713    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167716    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167719    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167722    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167725    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167728    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167733    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167736    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167739    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167742    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167746    2583 flags.go:64] FLAG: --enable-server="true"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167749    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167753    2583 flags.go:64] FLAG: --event-burst="100"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167756    2583 flags.go:64] FLAG: --event-qps="50"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167759    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167762    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167766    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 01 21:43:44.173158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167770    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167773    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167776    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167779    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167782    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167785    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167788    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167792    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167795    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167798    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167801    2583 flags.go:64] FLAG: --feature-gates=""
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167805    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167808    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167811    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167814    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167817    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167821    2583 flags.go:64] FLAG: --help="false"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167824    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167827    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167830    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167833    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167837    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167840    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 21:43:44.173854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167843    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167846    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167849    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167852    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167855    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167858    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167861    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167864    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167868    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167871    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167874    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167877    2583 flags.go:64] FLAG: --lock-file=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167880    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167882    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167885    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167890    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167893    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167897    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167900    2583 flags.go:64] FLAG: --logging-format="text"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167903    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167906    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167909    2583 flags.go:64] FLAG: --manifest-url=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167912    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167916    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167919    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 21:43:44.174413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167923    2583 flags.go:64] FLAG: --max-pods="110"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167926    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167929    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167932    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167935    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167938    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167941    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167944    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167951    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167954    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167957    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167960    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167964    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167967    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167970    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167973    2583 flags.go:64] FLAG: --port="10250"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167976    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167979    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05431e098173a1750"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167982    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167985    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167988    2583 flags.go:64] FLAG: --register-node="true"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167991    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167995    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.167997    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168000    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168004    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 01 21:43:44.175015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168008    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168011    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168014    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168017    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168020    2583 flags.go:64] FLAG: --runonce="false"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168023    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168026    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168029    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168032    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168035    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168038    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168041    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168044    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168046    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168051    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168054    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168056    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168060    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168063    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168066    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168071    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168074    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168077    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168081    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168084    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 21:43:44.175638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168087    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168091    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168094    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168096    2583 flags.go:64] FLAG: --v="2"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168100    2583 flags.go:64] FLAG: --version="false"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168104    2583 flags.go:64] FLAG: --vmodule=""
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168108    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168112    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168281    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168286    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168289    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168292    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168294    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168297    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168300    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168303    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168305    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168308    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168310    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168313    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168315    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.176237 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168319    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168321    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168324    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168327    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168329    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168332    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168334    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168337    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168340    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168342    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168344    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168347    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168349    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168352    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168354    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168356    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168359    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168361    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168364    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.176757 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168369    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168372    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168375    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168377    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168380    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168382    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168384    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168387    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168389    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168392    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168394    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168398    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168401    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168405    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168407    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168410    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168413    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168415    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168418    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.177332 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168422    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168426    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168430    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168433    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168435    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168438    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168440    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168443    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168445    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168448    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168450    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168453    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168456    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168458    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168461    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168463    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168466    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168469    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168471    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168474    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.177961 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168476    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168479    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168481    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168484    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168488    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168491    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168494    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168497    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168500    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168502    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168505    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168508    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168510    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168513    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168516    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168518    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168521    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168523    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168526    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168529    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.178787 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168531    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168534    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168536    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168539    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168542    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168544    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168547    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168549    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168552    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168556    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168559    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168562    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168565    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168567    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168570    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.179340 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.168594    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168628    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168633    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168638    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168641    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168644    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168647    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168649    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168654    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168669    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168673    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168676    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168679    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168681    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168684    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168686    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168689    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168691    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168694    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.179719 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168697    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168699    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168702    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168704    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168707    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168709    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168713    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168715    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168718    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168720    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168723    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168725    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168728    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168730    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168733    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168737    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168741    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168744    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168746    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168749    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.180167 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168752    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168754    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168758    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168761    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168764    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168767    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168769    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168772    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168774    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168777    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168779    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168782    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168785    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168787    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168790    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168793    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168795    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168797    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168801    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.180652 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168805    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168808    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168811    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168814    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168816    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168819    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168821    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168824    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168828    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168831    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168834    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168836    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168839    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168842    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168844    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168847    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168850    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168852    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168855    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.181117 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168857    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168860    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168863    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168865    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168868    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168870    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168873    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168875    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168878    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168880    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168883    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168885    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168888    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168890    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168893    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168896    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168899    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168901    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168903    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.181586 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168907    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168909    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168914    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168918    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168920    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168923    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168926    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168928    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168931    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168934    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.168936    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.175372    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.175388    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.176021    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.176028    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176080    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176085    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176087    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176090    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.182108 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176094    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176097    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176100    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176103    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176105    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176108    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176111    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176114    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176117    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176120    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176122    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176125    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176128    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176130    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176133    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176135    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176138    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176140    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176144    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.182574 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176148    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176151    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176154    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176157    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176160    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176162    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176166    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176169    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176172    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176174    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176177    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176179    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176182    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176184    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176187    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176189    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176192    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176195    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176197    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176200    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.183026 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176202    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176205    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176208    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176210    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176213    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176215    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176218    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176220    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176223    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176225    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176228    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176230    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176233    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176235    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176238    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176240    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176243    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176246    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176249    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.183545 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176253    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176255    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176258    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176261    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176264    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176266    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176269    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176271    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176274    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176277    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176279    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176282    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176284    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176287    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176289    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176292    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176295    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176297    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176300    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176302    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.184043 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176305    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176308    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176310    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176313    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176315    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176318    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176320    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176323    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176325    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176328    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176330    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176333    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176336    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176339    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176342    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176346    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176350    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176353    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176356    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.184517 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176358    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176361    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176363    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176366    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176368    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.176395    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176521    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176526    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176529    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176531    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176534    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176537    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176539    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176543    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176547    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176550    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.184976 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176552    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176555    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176558    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176561    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176563    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176566    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176568    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176571    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176574    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176576    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176579    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176582    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176584    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176587    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176590    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176593    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176595    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176598    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176600    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176602    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.185350 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176605    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176607    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176610    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176612    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176615    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176617    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176620    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176622    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176625    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176627    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176629    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176632    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176634    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176637    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176640    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176642    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176645    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176647    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176650    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176652    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.185889 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176655    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176671    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176675    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176678    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176683    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176685    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176689    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176693    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176696    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176699    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176702    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176704    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176707    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176710    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176713    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176716    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176718    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176721    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176723    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.186382 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176726    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176728    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176730    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176733    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176735    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176738    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176741    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176743    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176746    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176749    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176752    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176754    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176757    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176759    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176762    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176765    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176767    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176770    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176773    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.186843 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176776    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176779    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176782    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176784    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176787    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176789    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176792    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176794    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176798    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176800    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176803    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176806    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176808    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176811    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176813    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176816    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176818    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.187315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176821    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.176844    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176874    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176878    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176882    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176885    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176888    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176893    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176896    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176899    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176902    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176905    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176908    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176911    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176913    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176916    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:43:44.187745 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176919    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176921    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176924    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176926    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176929    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176931    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176934    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176936    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176940    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176942    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176945    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176948    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176950    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176953    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176955    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176958    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176960    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176963    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176965    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:43:44.188129 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176967    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176970    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176972    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176975    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176977    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176980    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176982    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176985    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176987    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176990    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176992    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176994    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.176997    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177000    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177002    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177005    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177007    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177010    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177012    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:43:44.188606 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177015    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177018    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177022    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177025    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177027    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177030    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177033    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177035    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177037    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177040    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177042    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177045    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177047    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177050    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177052    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177055    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177057    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177059    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177062    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:43:44.189068 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177064    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177067    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177069    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177072    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177074    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177077    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177079    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177082    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177084    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177087    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177089    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177092    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177094    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177097    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177099    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177102    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177104    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177107    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177110    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177112    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:43:44.189524 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177114    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177117    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177119    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177123    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177125    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177128    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177130    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177133    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177135    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177137    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177140    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177142    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177145    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177147    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:44.177150    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.177197    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.180342    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.181506    2583 server.go:1014] "Starting client certificate rotation"
Nov 01 21:43:44.190046 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.181602    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 21:43:44.190464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.182410    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 21:43:44.190464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.184232    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 21:43:44.190464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.188691    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 21:43:44.199768 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.199677    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 01 21:43:44.203456 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.203441    2583 log.go:25] "Validated CRI v1 image API"
Nov 01 21:43:44.204260 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.204241    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 21:43:44.206565 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.206547    2583 fs.go:135] Filesystem UUIDs: map[2e88aefb-6dad-4cdb-8af9-d5839ee96cb7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c24973aa-6c3b-4f8b-9211-63d0bac5a9ca:/dev/nvme0n1p4]
Nov 01 21:43:44.206616 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.206566    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 21:43:44.211956 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.211840    2583 manager.go:217] Machine: {Timestamp:2026-11-01 21:43:44.210575442 +0000 UTC m=+0.328308877 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099762 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261226253deb6643cf0ae29fc9a39e SystemUUID:ec261226-253d-eb66-43cf-0ae29fc9a39e BootID:57f795f9-fd02-4d78-a3e5-b4f48f806caa Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:88:ad:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:88:ad:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:f4:4f:67:90:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 21:43:44.211956 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.211953    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 21:43:44.212079 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.212022    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 21:43:44.215021 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215001    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 21:43:44.215192 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215022    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-146.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 21:43:44.215239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215196    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 21:43:44.215239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215204    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 21:43:44.215293 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215251    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 21:43:44.215293 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215269    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 21:43:44.215997 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.215987    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 21:43:44.219639 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219627    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 21:43:44.219765 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219755    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 21:43:44.219795 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219786    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 21:43:44.219821 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219795    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 21:43:44.219821 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219808    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 21:43:44.219821 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219816    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 01 21:43:44.219906 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.219825    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 21:43:44.220129 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.220118    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 21:43:44.222177 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.222152    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 21:43:44.222539 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.222508    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 21:43:44.226795 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.226781    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 21:43:44.227626 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227614    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 21:43:44.227674 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227644    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 21:43:44.227808 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227798    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 21:43:44.227849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227813    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 21:43:44.227849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227819    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 21:43:44.227849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227825    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 21:43:44.227849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227831    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 21:43:44.227849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.227836    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 21:43:44.228377 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228366    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 21:43:44.228411 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228382    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 21:43:44.228411 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228392    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 21:43:44.228411 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228402    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 21:43:44.228515 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228414    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 21:43:44.228515 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.228427    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 21:43:44.229612 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.229601    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 21:43:44.229708 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.229616    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 21:43:44.233388 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.233374    2583 server.go:1263] "Started kubelet"
Nov 01 21:43:44.233457 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.233432    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 21:43:44.234123 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.233930    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 21:43:44.234214 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.234144    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 21:43:44.234223 ip-10-0-135-146 systemd[1]: Started Kubernetes Kubelet.
Nov 01 21:43:44.234387 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.234285    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-146.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 21:43:44.235213 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.235197    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 21:43:44.235819 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.235794    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 21:43:44.239437 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.239417    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 21:43:44.239530 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.239510    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-146.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 21:43:44.239575 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.239507    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 21:43:44.239865 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.239847    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 21:43:44.239979 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.238850    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-146.us-east-2.compute.internal.18e409e37b6e2c39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-146.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-146.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 21:43:44.233344057 +0000 UTC m=+0.351077491,LastTimestamp:2026-11-01 21:43:44.233344057 +0000 UTC m=+0.351077491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-146.us-east-2.compute.internal,}"
Nov 01 21:43:44.239979 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.239933    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.240482 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.240442    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 21:43:44.240482 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240469    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 21:43:44.240611 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240490    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 21:43:44.240611 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240455    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 21:43:44.240739 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240627    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 21:43:44.240739 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240634    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 21:43:44.240823 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240780    2583 factory.go:55] Registering systemd factory
Nov 01 21:43:44.240823 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.240795    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 01 21:43:44.241047 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.241031    2583 factory.go:153] Registering CRI-O factory
Nov 01 21:43:44.241103 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.241050    2583 factory.go:223] Registration of the crio container factory successfully
Nov 01 21:43:44.241103 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.241100    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 21:43:44.241178 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.241119    2583 factory.go:103] Registering Raw factory
Nov 01 21:43:44.241178 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.241138    2583 manager.go:1199] Started watching for new ooms in manager
Nov 01 21:43:44.242016 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.242003    2583 manager.go:319] Starting recovery of all containers
Nov 01 21:43:44.242827 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.242807    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fjkc"
Nov 01 21:43:44.251298 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.251201    2583 manager.go:324] Recovery completed
Nov 01 21:43:44.254084 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.254050    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-146.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 21:43:44.254211 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.254188    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9fjkc"
Nov 01 21:43:44.255619 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.255593    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 21:43:44.258316 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.258301    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.260061 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260044    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.260150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260076    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.260150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260091    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.260633 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260616    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 01 21:43:44.260633 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260632    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 21:43:44.260776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260653    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 21:43:44.260776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.260635    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.261567 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.261550    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.261624 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.261581    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.261624 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.261594    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.341065 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.341045    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.404168 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.404153    2583 policy_none.go:50] "Start"
Nov 01 21:43:44.404238 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.404172    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 21:43:44.404238 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.404184    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 21:43:44.406286 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.406275    2583 policy_none.go:44] "Start"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.444507    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.446275    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.448807    2583 manager.go:341] "Starting Device Plugin manager"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.448844    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.448855    2583 server.go:87] "Starting device plugin registration server"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.449221    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.449233    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.449337    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.449421    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.449429    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.450043    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 21:43:44.460397 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.450078    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.462706 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.461046    2583 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-146.us-east-2.compute.internal\" not found" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.481746 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.481724    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 21:43:44.482946 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.482932    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 21:43:44.483033 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.482951    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 21:43:44.483033 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.482967    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 21:43:44.483033 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.483002    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 21:43:44.486551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.486538    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:43:44.550084 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.550066    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.557578 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.557561    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.557654 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.557586    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.557654 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.557601    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.557654 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.557628    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.566478 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.566464    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.566560 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.566483    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-146.us-east-2.compute.internal\": node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.580340 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.580323    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.583423 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.583405    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"]
Nov 01 21:43:44.583517 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.583453    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.587469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.587452    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.587523 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.587481    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.587523 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.587491    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.589619 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.589607    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.589775 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.589759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.589837 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.589794    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.590712 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590697    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.590777 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590712    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.590777 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590732    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.590777 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590738    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.590777 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590745    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.590777 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.590749    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.592784 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.592767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.592854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.592793    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:43:44.593582 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.593566    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:43:44.593690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.593598    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:43:44.593690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.593609    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:43:44.610215 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.610193    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.616531 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.616516    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.643740 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.643717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.643828 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.643749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.643828 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.643782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.680639 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.680601    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.744064 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.744131 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.744131 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.744131 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744117    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.744229 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744130    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.744229 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.744140    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.781188 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.781167    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.881723 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.881696    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:44.912854 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.912825    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.920506 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:44.920484    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:44.982402 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:44.982340    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:45.082907 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:45.082875    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.254963 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.254944    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 21:43:46.255278 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.255063    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 21:43:46.257091 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.257075    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.333019 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.332950    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:45 +0000 UTC" deadline="2028-05-28 16:25:00.636261621 +0000 UTC"
Nov 01 21:43:46.333019 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.332976    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13770h41m14.303289582s"
Nov 01 21:43:46.357716 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.357690    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.458244 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.458216    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.558336 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.558317    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.659065 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.659034    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.683992 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.683969    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:43:46.733535 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:46.733323    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181995cb0fabc637e7891e096db17634.slice/crio-c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364 WatchSource:0}: Error finding container c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364: Status 404 returned error can't find the container with id c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364
Nov 01 21:43:46.733730 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:46.733711    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec31c2ba0ee2bf174eddd7fce91e152b.slice/crio-5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348 WatchSource:0}: Error finding container 5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348: Status 404 returned error can't find the container with id 5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348
Nov 01 21:43:46.739764 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.739749    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:43:46.759790 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.759747    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.774809 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.773445    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:43:46.860586 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:46.860542    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:43:46.897542 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.897521    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:43:46.914607 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.914578    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:46.927368 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.927344    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:43:46.928567 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.928551    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:43:46.935980 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:46.935964    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:43:47.294565 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.294541    2583 apiserver.go:52] "Watching apiserver"
Nov 01 21:43:47.301481 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.301449    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 21:43:47.301778 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.301758    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-r7hr8","kube-system/konnectivity-agent-9wtgk","openshift-multus/multus-8gkcc","openshift-multus/network-metrics-daemon-bcxfp","openshift-network-diagnostics/network-check-target-f5s44","openshift-ovn-kubernetes/ovnkube-node-zw7wh","kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-xs9kv","openshift-network-operator/iptables-alerter-bdpxm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-f4s8z","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2","openshift-cluster-node-tuning-operator/tuned-6wp5h","openshift-dns/node-resolver-r86vn"]
Nov 01 21:43:47.305477 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.305456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.307521 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.307505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.309142 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.309123    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.309532 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.309511    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.309635 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.309600    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\""
Nov 01 21:43:47.309635 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.309617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.310890 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.310870    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\""
Nov 01 21:43:47.310890 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.310884    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 21:43:47.311092 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.311073    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 21:43:47.311971 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.311954    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.312053 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.312021    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:47.312881 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.312866    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 21:43:47.312951 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.312939    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.312994 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.312941    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.312994 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.312982    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\""
Nov 01 21:43:47.313135 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.313125    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 21:43:47.313717 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.313705    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 21:43:47.314052 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.314037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:47.314136 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.314092    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:47.316273 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.316259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.318601 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.318585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.320795 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.320778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.324563 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.324542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.325323 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.325407 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.325407 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.325407 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.325407 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.325407 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.325645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.325645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.325645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.325645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.325645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.325880 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.325971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmp4t\" (UniqueName: \"kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.326112 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv58b\" (UniqueName: \"kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57cl\" (UniqueName: \"kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326704    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.326719 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5kx\" (UniqueName: \"kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.326999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.327054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.327210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.327094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.327660 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.327169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.327660 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.327247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.328496 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.328479    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 21:43:47.329560 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.329540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.329646 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.329611    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:47.332278 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.332259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.348337 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.348271    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-62656"
Nov 01 21:43:47.358263 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.358238    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-62656"
Nov 01 21:43:47.410781 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410264    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.410781 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410371    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.410781 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410499    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.410781 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410623    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 21:43:47.410781 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410694    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 21:43:47.411133 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410904    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 21:43:47.411133 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.410991    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 21:43:47.411133 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411062    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 21:43:47.411133 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411070    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.411133 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411126    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.411370 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411140    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.411370 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411171    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 21:43:47.411370 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411182    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.411370 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411262    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.411606 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411577    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 21:43:47.411742 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411691    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\""
Nov 01 21:43:47.411742 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411699    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\""
Nov 01 21:43:47.411742 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411722    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 21:43:47.411899 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411782    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 21:43:47.411899 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411822    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\""
Nov 01 21:43:47.411899 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411803    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\""
Nov 01 21:43:47.412015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411901    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 21:43:47.412015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411923    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\""
Nov 01 21:43:47.412015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.411987    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\""
Nov 01 21:43:47.412100 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.412088    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 21:43:47.416063 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.416044    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 21:43:47.427891 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.427860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.427988 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.427901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.427988 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.427957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.427988 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.427978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.428156 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428051    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428156 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428057    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.428260 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.428330 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.428330 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428169    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.428330 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428211    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.428476 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57cl\" (UniqueName: \"kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.428476 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428476 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428476 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428465    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976g5\" (UniqueName: \"kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428600    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.428673 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428621    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5kx\" (UniqueName: \"kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428901    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.428950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428959    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428969    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.428976    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429231 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429268    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429396    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746xf\" (UniqueName: \"kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429552    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429551    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429664    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429686    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.429776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429705    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc56\" (UniqueName: \"kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.429994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430025    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430111    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.430315 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430132    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430106    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430154    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430195    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfrk\" (UniqueName: \"kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.430274    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430337    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.430348    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:47.930323384 +0000 UTC m=+2.974421106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9n7x\" (UniqueName: \"kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430391    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430495    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.431086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430569    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430958    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431012    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.430991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431078    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp4t\" (UniqueName: \"kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431350    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.431860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431397    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431401    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431443    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431463    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv58b\" (UniqueName: \"kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431546    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431615    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431678    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.432580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431731    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431814    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:47.433353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.431877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.435156 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.435131    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.435156 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.435146    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.435345 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.435329    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.435530 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.435502    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.440678 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.440650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57cl\" (UniqueName: \"kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.441167 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.441145    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5kx\" (UniqueName: \"kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.441264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.441183    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:43:47.441264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.441207    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:43:47.441264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.441219    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:47.441440 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.441290    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:43:47.941272956 +0000 UTC m=+2.985370673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:47.443737 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.443716    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv58b\" (UniqueName: \"kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.443830 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.443724    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp4t\" (UniqueName: \"kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.532375 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976g5\" (UniqueName: \"kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746xf\" (UniqueName: \"kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.532551 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc56\" (UniqueName: \"kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfrk\" (UniqueName: \"kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532789    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9n7x\" (UniqueName: \"kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.532999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533369    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533566    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.532455    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533616    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.533682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533651    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.534104 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.533840    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.534157 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534142    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534205    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534249    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534326    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534344    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534401    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.534483    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.534544    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:48.034526014 +0000 UTC m=+3.078623718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:47.535002 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.534945    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.535347 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.535021    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.535588 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.535564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.535699 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.535633    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.545051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.544973    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976g5\" (UniqueName: \"kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.545609 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.545589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc56\" (UniqueName: \"kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.545707 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.545684    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9n7x\" (UniqueName: \"kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.547170 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.547150    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfrk\" (UniqueName: \"kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.547660 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.547638    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746xf\" (UniqueName: \"kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.561460 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.561421    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerStarted","Data":"c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364"}
Nov 01 21:43:47.562744 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.562724    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"ec31c2ba0ee2bf174eddd7fce91e152b","Type":"ContainerStarted","Data":"5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348"}
Nov 01 21:43:47.578078 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.578054    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:43:47.617385 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.617348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:43:47.626187 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.626162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:47.632665 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.632635    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef221e0c_af12_43b6_9102_4a68591df51f.slice/crio-8b55f9c582fc947f0052507ac1ae3a5014c9a5abc86750f375e3f146895ae0aa WatchSource:0}: Error finding container 8b55f9c582fc947f0052507ac1ae3a5014c9a5abc86750f375e3f146895ae0aa: Status 404 returned error can't find the container with id 8b55f9c582fc947f0052507ac1ae3a5014c9a5abc86750f375e3f146895ae0aa
Nov 01 21:43:47.633976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.633787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8gkcc"
Nov 01 21:43:47.638084 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.638060    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac098ba_33a8_4a99_812f_ae5561684d7c.slice/crio-089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1 WatchSource:0}: Error finding container 089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1: Status 404 returned error can't find the container with id 089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1
Nov 01 21:43:47.647796 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.647761    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b8c97d_537b_46e4_aa07_339c0d0b225b.slice/crio-95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9 WatchSource:0}: Error finding container 95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9: Status 404 returned error can't find the container with id 95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9
Nov 01 21:43:47.659097 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.659071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:43:47.663964 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.663945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:43:47.666471 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.666431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:43:47.671232 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.671213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:43:47.671839 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.671771    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d602079_f8c5_49f7_afe1_8d6e074b8c27.slice/crio-88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b WatchSource:0}: Error finding container 88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b: Status 404 returned error can't find the container with id 88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b
Nov 01 21:43:47.679315 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.679135    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed0239d_76c2_4d3b_b168_560c022e4e3a.slice/crio-0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7 WatchSource:0}: Error finding container 0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7: Status 404 returned error can't find the container with id 0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7
Nov 01 21:43:47.682081 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.682019    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0aef984_8fc9_46e0_8243_27b9935991a3.slice/crio-defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a WatchSource:0}: Error finding container defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a: Status 404 returned error can't find the container with id defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a
Nov 01 21:43:47.682570 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.682546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:43:47.682981 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.682957    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf3bf5f_cbda_43b8_b263_dad640f9f5a3.slice/crio-fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228 WatchSource:0}: Error finding container fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228: Status 404 returned error can't find the container with id fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228
Nov 01 21:43:47.684658 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.684596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:43:47.698989 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.698957    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d867fa_5c9f_4cab_ae4e_08f4e3296a81.slice/crio-0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8 WatchSource:0}: Error finding container 0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8: Status 404 returned error can't find the container with id 0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8
Nov 01 21:43:47.700973 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:43:47.700950    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dd454d_3bb0_48d9_b6ed_d907da241372.slice/crio-90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56 WatchSource:0}: Error finding container 90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56: Status 404 returned error can't find the container with id 90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56
Nov 01 21:43:47.936371 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:47.936278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:47.936534 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.936429    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:47.936534 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:47.936508    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:48.936489861 +0000 UTC m=+3.980587559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:48.037619 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.037582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:48.037767 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.037636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:48.037767 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037707    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:48.037876 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037784    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:43:48.037876 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037804    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:43:48.037876 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037807    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:49.03776473 +0000 UTC m=+4.081862445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:48.037876 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037817    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:48.037876 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.037853    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:43:49.037842413 +0000 UTC m=+4.081940128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:48.361517 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.361450    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:47 +0000 UTC" deadline="2028-07-01 06:38:01.625134359 +0000 UTC"
Nov 01 21:43:48.361517 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.361515    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14576h54m13.263623799s"
Nov 01 21:43:48.557068 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.557032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:48.557224 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.557177    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:48.557622 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.557601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:48.557726 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.557706    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:48.568051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.567957    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wp5h" event={"ID":"ef221e0c-af12-43b6-9102-4a68591df51f","Type":"ContainerStarted","Data":"8b55f9c582fc947f0052507ac1ae3a5014c9a5abc86750f375e3f146895ae0aa"}
Nov 01 21:43:48.571283 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.571198    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerStarted","Data":"90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56"}
Nov 01 21:43:48.573500 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.573474    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r86vn" event={"ID":"baf3bf5f-cbda-43b8-b263-dad640f9f5a3","Type":"ContainerStarted","Data":"fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228"}
Nov 01 21:43:48.576437 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.576384    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdpxm" event={"ID":"f0aef984-8fc9-46e0-8243-27b9935991a3","Type":"ContainerStarted","Data":"defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a"}
Nov 01 21:43:48.579186 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.579161    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8gkcc" event={"ID":"40b8c97d-537b-46e4-aa07-339c0d0b225b","Type":"ContainerStarted","Data":"95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9"}
Nov 01 21:43:48.581188 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.581119    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerStarted","Data":"0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8"}
Nov 01 21:43:48.587338 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.584512    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7"}
Nov 01 21:43:48.590099 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.590016    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7hr8" event={"ID":"8d602079-f8c5-49f7-afe1-8d6e074b8c27","Type":"ContainerStarted","Data":"88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b"}
Nov 01 21:43:48.592074 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.592008    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wtgk" event={"ID":"dac098ba-33a8-4a99-812f-ae5561684d7c","Type":"ContainerStarted","Data":"089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1"}
Nov 01 21:43:48.950347 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:48.948873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:48.950347 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.949085    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:48.950347 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:48.949150    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:50.949128758 +0000 UTC m=+5.993226455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:49.049481 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.049446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:49.049601 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.049504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:49.049734 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049716    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:43:49.049813 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049741    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:43:49.049813 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049754    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:49.049914 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049834    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:43:51.049789134 +0000 UTC m=+6.093886844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:49.049914 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049910    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:49.050029 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.049948    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:51.049935418 +0000 UTC m=+6.094033129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:49.362435 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.362366    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:47 +0000 UTC" deadline="2028-04-12 03:32:57.53564627 +0000 UTC"
Nov 01 21:43:49.362435 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.362405    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12653h49m8.173245848s"
Nov 01 21:43:49.557483 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.557455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:49.557658 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:49.557612    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:49.603292 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.603254    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"ec31c2ba0ee2bf174eddd7fce91e152b","Type":"ContainerStarted","Data":"56240583552aaef3ef5aa70d5de9b7b0633f3fa081072db778f49b9e89996bd9"}
Nov 01 21:43:49.620205 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:49.619570    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal" podStartSLOduration=3.619551495 podStartE2EDuration="3.619551495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:49.619050238 +0000 UTC m=+4.663147952" watchObservedRunningTime="2026-11-01 21:43:49.619551495 +0000 UTC m=+4.663649215"
Nov 01 21:43:50.557604 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:50.557572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:50.558070 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:50.557718    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:50.558242 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:50.558133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:50.558351 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:50.558282    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:50.970402 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:50.970136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:50.970682 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:50.970622    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:50.970769 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:50.970709    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:43:54.970688529 +0000 UTC m=+10.014786244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.071396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.071453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.071593    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.071624    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.071638    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.071694    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.071676636 +0000 UTC m=+10.115774347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.072101    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:51.072189 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.072153    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:43:55.07213762 +0000 UTC m=+10.116235380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:51.557575 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.557546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:51.557751 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:51.557721    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:51.615525 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.615482    2583 generic.go:356] "Generic (PLEG): container finished" podID="181995cb0fabc637e7891e096db17634" containerID="41b22f885ac1fbddb720e1994bfe4662d833d63837ed65077708d9d20ba66329" exitCode=0
Nov 01 21:43:51.615696 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.615570    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerDied","Data":"41b22f885ac1fbddb720e1994bfe4662d833d63837ed65077708d9d20ba66329"}
Nov 01 21:43:51.621518 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.620835    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="eb98c7e91d0609f94f256ae6da38a440d7d2c409b4d79d77a5e1cc818635c216" exitCode=0
Nov 01 21:43:51.621518 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:51.620878    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"eb98c7e91d0609f94f256ae6da38a440d7d2c409b4d79d77a5e1cc818635c216"}
Nov 01 21:43:52.557273 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.557237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:52.557470 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:52.557385    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:52.557470 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.557451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:52.557581 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:52.557529    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:52.629292 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.629258    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerStarted","Data":"59eb7bcd2e9f23dac493b9d099e328109074a84d3ee33c1af29b59aeadd783bc"}
Nov 01 21:43:52.645369 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.644844    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wtgk" event={"ID":"dac098ba-33a8-4a99-812f-ae5561684d7c","Type":"ContainerStarted","Data":"2f5e4b11e947b28ef0e2a087db18fcfbaf2548cff4bcdc1962176f0e08192a3d"}
Nov 01 21:43:52.648062 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.646980    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" podStartSLOduration=6.64696398 podStartE2EDuration="6.64696398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:52.646617544 +0000 UTC m=+7.690715265" watchObservedRunningTime="2026-11-01 21:43:52.64696398 +0000 UTC m=+7.691061700"
Nov 01 21:43:52.676632 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:52.676565    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9wtgk" podStartSLOduration=3.693797895 podStartE2EDuration="7.676537987s" totalImagesPullingTime="3.982740092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:52.674310696 +0000 UTC m=+7.718408407" watchObservedRunningTime="2026-11-01 21:43:52.676537987 +0000 UTC m=+7.720635713"
Nov 01 21:43:53.326594 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:53.326564    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:53.327347 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:53.327320    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:43:53.563156 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:53.563126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:53.563327 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:53.563309    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:53.653366 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:53.650324    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7hr8" event={"ID":"8d602079-f8c5-49f7-afe1-8d6e074b8c27","Type":"ContainerStarted","Data":"390dec5b2e8bb188d18d0c851be56ded0605908558cfa424ee2ac78d71f4f108"}
Nov 01 21:43:53.668243 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:53.668188    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r7hr8" podStartSLOduration=3.546373001 podStartE2EDuration="8.668169365s" totalImagesPullingTime="5.121796364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:53.667429628 +0000 UTC m=+8.711527346" watchObservedRunningTime="2026-11-01 21:43:53.668169365 +0000 UTC m=+8.712267085"
Nov 01 21:43:54.557875 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:54.557839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:54.558064 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:54.557973    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:54.558383 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:54.558351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:54.558489 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:54.558453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:54.662469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:54.662426    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerStarted","Data":"2cfd9a8996e696a5c2e1af6dad43dd2ca3312f1a4d84b973237769ceb9125624"}
Nov 01 21:43:54.662851 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:54.662527    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 21:43:55.007356 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.006993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:55.007356 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.007155    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:55.007356 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.007224    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:44:03.007206644 +0000 UTC m=+18.051304345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:43:55.107889 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.107846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:55.108049 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.107965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:55.108103 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108090    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:55.108170 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108153    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:03.108135399 +0000 UTC m=+18.152233109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:43:55.108257 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108241    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:43:55.108324 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108259    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:43:55.108324 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108272    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:55.108389 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.108325    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:44:03.108312311 +0000 UTC m=+18.152410026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:43:55.213629 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.213602    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 21:43:55.529775 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.528960    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:43:55.213623704Z","UUID":"e40703f2-4a16-41e7-9277-722fd332199c","Handler":null,"Name":"","Endpoint":""}
Nov 01 21:43:55.532714 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.531639    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 21:43:55.532714 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.531669    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 21:43:55.557150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.557123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:55.557321 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:55.557262    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:55.667509 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.667475    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerStarted","Data":"4137415e9cf02c01853295709a020664e4caa0d5837ec3421318d6c4cbe827ee"}
Nov 01 21:43:55.672377 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.672249    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8gkcc" event={"ID":"40b8c97d-537b-46e4-aa07-339c0d0b225b","Type":"ContainerStarted","Data":"87b78e11761fc2cd1a5c18256f00eb575a0878a4f9812308f9ae19902648df48"}
Nov 01 21:43:55.694975 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:55.693982    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8gkcc" podStartSLOduration=2.933683252 podStartE2EDuration="10.693965749s" totalImagesPullingTime="7.760282497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:55.69362069 +0000 UTC m=+10.737718408" watchObservedRunningTime="2026-11-01 21:43:55.693965749 +0000 UTC m=+10.738063469"
Nov 01 21:43:56.557041 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.556918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:56.557150 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:56.557040    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:56.557150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.557094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:56.557221 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:56.557188    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:56.676127 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.676091    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="1cb0b6e0ec28eeb7c46cbc1e33a9a526d11b32fa776036d36117575441684a3d" exitCode=0
Nov 01 21:43:56.676569 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.676175    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"1cb0b6e0ec28eeb7c46cbc1e33a9a526d11b32fa776036d36117575441684a3d"}
Nov 01 21:43:56.678913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.678879    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerStarted","Data":"18b3a0c6099c392275bcb765090ac0e7ce538a1af75181187b97bd95c4aa5888"}
Nov 01 21:43:56.680456 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.680426    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r86vn" event={"ID":"baf3bf5f-cbda-43b8-b263-dad640f9f5a3","Type":"ContainerStarted","Data":"2fb8383666eedb8b8c04260a8fd1839c856abeac2f883023e3adacc34028b87f"}
Nov 01 21:43:56.717726 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.717685    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r86vn" podStartSLOduration=3.753670339 podStartE2EDuration="11.717669693s" totalImagesPullingTime="7.963999354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:56.717601973 +0000 UTC m=+11.761699692" watchObservedRunningTime="2026-11-01 21:43:56.717669693 +0000 UTC m=+11.761767413"
Nov 01 21:43:56.737783 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:56.737745    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podStartSLOduration=3.655458123 podStartE2EDuration="11.73773488s" totalImagesPullingTime="8.082276757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:56.737441738 +0000 UTC m=+11.781539456" watchObservedRunningTime="2026-11-01 21:43:56.73773488 +0000 UTC m=+11.781832599"
Nov 01 21:43:57.559748 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.559721    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:57.559862 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:57.559814    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:43:57.684375 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.684196    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerStarted","Data":"6cf87a8956cb1565ec7858fbd967961b3f9c40ddd15fd10bc9db5b1287d308ed"}
Nov 01 21:43:57.686003 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.685982    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6wp5h" event={"ID":"ef221e0c-af12-43b6-9102-4a68591df51f","Type":"ContainerStarted","Data":"142100e8b73e8b9ac835df5f961b7a2455b8d41f58d3407b4482918f32f66fad"}
Nov 01 21:43:57.687523 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.687501    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdpxm" event={"ID":"f0aef984-8fc9-46e0-8243-27b9935991a3","Type":"ContainerStarted","Data":"91613f7b27dd93371bebb0e3764bab8fd390538a88c1f8a3e794042a86617755"}
Nov 01 21:43:57.745445 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.745407    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bdpxm" podStartSLOduration=4.671728401 podStartE2EDuration="12.745395743s" totalImagesPullingTime="8.073667342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:57.726737047 +0000 UTC m=+12.770834765" watchObservedRunningTime="2026-11-01 21:43:57.745395743 +0000 UTC m=+12.789493462"
Nov 01 21:43:57.745581 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:57.745564    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6wp5h" podStartSLOduration=3.033040884 podStartE2EDuration="12.745560804s" totalImagesPullingTime="9.71251992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:43:57.745330113 +0000 UTC m=+12.789427829" watchObservedRunningTime="2026-11-01 21:43:57.745560804 +0000 UTC m=+12.789658524"
Nov 01 21:43:58.557052 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:58.557023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:43:58.557190 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:58.557023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:43:58.557190 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:58.557130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:43:58.557267 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:58.557187    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:43:58.691543 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:58.691499    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="6cf87a8956cb1565ec7858fbd967961b3f9c40ddd15fd10bc9db5b1287d308ed" exitCode=0
Nov 01 21:43:58.692017 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:58.691557    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"6cf87a8956cb1565ec7858fbd967961b3f9c40ddd15fd10bc9db5b1287d308ed"}
Nov 01 21:43:59.559695 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:43:59.559663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:43:59.559866 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:43:59.559760    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:00.557010 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:00.556985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:00.557788 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:00.556985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:00.557788 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:00.557091    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:00.557788 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:00.557187    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:00.697723 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:00.697677    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="65c2099eb421bb5f417d55637a21a08d412b1c2dbf93356f9e5dc7b32965101b" exitCode=0
Nov 01 21:44:00.697723 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:00.697716    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"65c2099eb421bb5f417d55637a21a08d412b1c2dbf93356f9e5dc7b32965101b"}
Nov 01 21:44:01.561017 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:01.560986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:01.561454 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:01.561123    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:02.557521 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:02.557491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:02.557681 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:02.557592    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:02.557681 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:02.557643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:02.557750 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:02.557719    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:03.071218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.071157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:03.071554 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.071265    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:44:03.071554 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.071335    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:44:19.071321878 +0000 UTC m=+34.115419576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:44:03.172195 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.172166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:03.172343 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.172201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:03.172343 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172293    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:44:03.172343 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172327    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:44:03.172343 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172342    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:44:03.172462 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172351    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:44:03.172462 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172372    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:19.172359262 +0000 UTC m=+34.216456959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:44:03.172462 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.172387    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:44:19.172381349 +0000 UTC m=+34.216479046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:44:03.560464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.560437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:03.560619 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:03.560571    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:03.706012 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.705916    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="0afc435f966c9e5f50b1cccea8a74c99ce85b90273c6cef15b1e904c53e99b0b" exitCode=0
Nov 01 21:44:03.706012 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:03.705966    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"0afc435f966c9e5f50b1cccea8a74c99ce85b90273c6cef15b1e904c53e99b0b"}
Nov 01 21:44:04.557362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:04.557334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:04.557362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:04.557360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:04.557746 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:04.557421    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:04.557746 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:04.557475    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:04.711569 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:04.711536    2583 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="3dee586a2835bd1671154f6f0ff5b788f6a446eb3cc6c4eed2e35cf8bb01bc7c" exitCode=0
Nov 01 21:44:04.711726 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:04.711602    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerDied","Data":"3dee586a2835bd1671154f6f0ff5b788f6a446eb3cc6c4eed2e35cf8bb01bc7c"}
Nov 01 21:44:05.000400 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:05.000206    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r86vn_baf3bf5f-cbda-43b8-b263-dad640f9f5a3/dns-node-resolver/0.log"
Nov 01 21:44:05.563497 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:05.563467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:05.563949 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:05.563614    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:05.717505 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:05.717472    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" event={"ID":"43d867fa-5c9f-4cab-ae4e-08f4e3296a81","Type":"ContainerStarted","Data":"c42a935e9a0742ca47a3a85d425671d85f704196bef1feb9ee4878efcb166054"}
Nov 01 21:44:05.974489 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:05.974462    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7hr8_8d602079-f8c5-49f7-afe1-8d6e074b8c27/node-ca/0.log"
Nov 01 21:44:06.557565 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:06.557529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:06.557730 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:06.557542    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:06.557730 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:06.557655    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:06.557830 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:06.557749    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:07.559940 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:07.559916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:07.560344 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:07.560029    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:08.556825 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:08.556789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:08.556825 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:08.556823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:08.557021 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:08.556903    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:08.557056 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:08.557026    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:09.085846 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.085819    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:09.086381 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.085939    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 21:44:09.086381 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.086348    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:09.102635 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.102598    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xs9kv" podStartSLOduration=13.283236124 podStartE2EDuration="24.102584864s" totalImagesPullingTime="10.81934874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:05.743465636 +0000 UTC m=+20.787563354" watchObservedRunningTime="2026-11-01 21:44:09.102584864 +0000 UTC m=+24.146682577"
Nov 01 21:44:09.556917 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.556899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:09.557030 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:09.557003    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:09.728578 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.728554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:09.728897 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.728876    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ed0239d-76c2-4d3b-b168-560c022e4e3a" containerID="1a4bc2f8b6dca409e57fd2921b947db7a6769abfa06e54b8735fd969760d8e0f" exitCode=1
Nov 01 21:44:09.728956 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.728944    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"6da6e871b5c2b7fe3bc3f3f9e73347b2f62b3a4fba44d29b3d1c215cb734400e"}
Nov 01 21:44:09.729015 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.729002    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"46ecf4635eb7a863cdbc5a495d885575b655fae97d07393898361c3117691d34"}
Nov 01 21:44:09.729053 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.729045    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"8061144bede762053683553dcef5452ba50aaaf47e920e8e8e524508f57a5677"}
Nov 01 21:44:09.729082 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.729076    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"5069819dc240a0acdae19004319a5d61490008a03aed8cebffac5afa69920531"}
Nov 01 21:44:09.729121 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.729111    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerDied","Data":"1a4bc2f8b6dca409e57fd2921b947db7a6769abfa06e54b8735fd969760d8e0f"}
Nov 01 21:44:09.729150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:09.729144    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"5056ca18b895e8e4209f3e38b8d47cdbb36ad240ac294834e44254e90c6a4551"}
Nov 01 21:44:10.557436 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:10.557407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:10.557802 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:10.557407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:10.557802 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:10.557512    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:10.557802 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:10.557574    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:11.557703 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:11.557672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:11.558070 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:11.557770    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:11.736620 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:11.736597    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:11.737021 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:11.736990    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"3d7696512570637b547bb7ce2955b792c69bce91ac60788a1d089c17bf2486f8"}
Nov 01 21:44:12.557335 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:12.557290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:12.557335 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:12.557338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:12.557587 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:12.557407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:12.557587 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:12.557522    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:13.557660 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:13.557633    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:13.558064 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:13.557733    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:14.557005 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.556974    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:14.557157 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:14.557085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:14.557157 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.557137    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:14.557234 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:14.557216    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:14.746036 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.746008    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:14.746818 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.746344    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"08e7dc7272c1916c7286790a40e3caa2e1d39c076bfc1a57c40c25deb3e6cd90"}
Nov 01 21:44:14.746818 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.746667    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:14.746818 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.746698    2583 scope.go:122] "RemoveContainer" containerID="1a4bc2f8b6dca409e57fd2921b947db7a6769abfa06e54b8735fd969760d8e0f"
Nov 01 21:44:14.762118 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:14.762098    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:15.558173 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.558140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:15.558341 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:15.558230    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:15.751888 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.751863    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:15.752293 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.752168    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" event={"ID":"2ed0239d-76c2-4d3b-b168-560c022e4e3a","Type":"ContainerStarted","Data":"1df06af0a4c71bcc156857137482b71389bc9fed495e6368652c1242ea80374c"}
Nov 01 21:44:15.752465 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.752444    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:15.752534 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.752522    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:15.769940 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.769919    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:15.793459 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:15.793422    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" podStartSLOduration=9.336032865 podStartE2EDuration="30.793411985s" totalImagesPullingTime="21.45737912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:15.792279462 +0000 UTC m=+30.836377181" watchObservedRunningTime="2026-11-01 21:44:15.793411985 +0000 UTC m=+30.837509704"
Nov 01 21:44:16.557483 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:16.557456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:16.557640 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:16.557456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:16.557640 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:16.557539    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:16.557640 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:16.557618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:17.559424 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:17.559392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:17.559817 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:17.559486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:18.557014 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:18.556982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:18.557170 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:18.557091    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:18.557170 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:18.557145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:18.557252 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:18.557235    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:19.093265 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:19.093235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:19.093645 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.093364    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:44:19.093645 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.093417    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs podName:bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b nodeName:}" failed. No retries permitted until 2026-11-01 21:44:51.093399907 +0000 UTC m=+66.137497619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs") pod "network-metrics-daemon-bcxfp" (UID: "bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 21:44:19.194093 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:19.194067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:19.194093 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:19.194098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:19.194264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194188    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 21:44:19.194264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194202    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 21:44:19.194264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194219    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 21:44:19.194264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194229    2583 projected.go:196] Error preparing data for projected volume kube-api-access-4zqlt for pod openshift-network-diagnostics/network-check-target-f5s44: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:44:19.194264 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194251    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret podName:c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61 nodeName:}" failed. No retries permitted until 2026-11-01 21:44:51.194232062 +0000 UTC m=+66.238329776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret") pod "global-pull-secret-syncer-f4s8z" (UID: "c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61") : object "kube-system"/"original-pull-secret" not registered
Nov 01 21:44:19.194434 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.194267    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt podName:03eefce5-3634-456f-889c-3ea91faaff0d nodeName:}" failed. No retries permitted until 2026-11-01 21:44:51.194259052 +0000 UTC m=+66.238356749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zqlt" (UniqueName: "kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt") pod "network-check-target-f5s44" (UID: "03eefce5-3634-456f-889c-3ea91faaff0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 21:44:19.560288 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:19.560258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:19.560447 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:19.560376    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:20.557434 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:20.557405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:20.557434 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:20.557430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:20.557844 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:20.557502    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:20.557844 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:20.557618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:21.559819 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:21.559793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:21.560181 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:21.559888    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:22.557380 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:22.557351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:22.557533 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:22.557351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:22.557533 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:22.557439    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:22.557602 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:22.557514    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:23.557746 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:23.557719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:23.558121 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:23.557815    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:24.460694 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.460619    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4s8z"]
Nov 01 21:44:24.460887 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.460787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:24.460943 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:24.460892    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:24.463950 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.463925    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5s44"]
Nov 01 21:44:24.464204 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.464190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:24.464418 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:24.464398    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:24.464525 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.464499    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bcxfp"]
Nov 01 21:44:24.464699 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:24.464673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:24.464812 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:24.464771    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:26.557645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:26.557613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:26.557645 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:26.557640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:26.558151 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:26.557613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:26.558151 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:26.557719    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:26.558151 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:26.557780    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:26.558151 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:26.557860    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:28.557046 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:28.557015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:28.557398 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:28.557043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:28.557398 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:28.557129    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcxfp" podUID="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b"
Nov 01 21:44:28.557398 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:28.557165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:28.557398 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:28.557225    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5s44" podUID="03eefce5-3634-456f-889c-3ea91faaff0d"
Nov 01 21:44:28.557398 ip-10-0-135-146 kubenswrapper[2583]: E1101 21:44:28.557282    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f4s8z" podUID="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61"
Nov 01 21:44:29.203030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.202956    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeReady"
Nov 01 21:44:29.203185 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.203088    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 21:44:29.252355 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.252323    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bgzv8"]
Nov 01 21:44:29.254229 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.254207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.255381 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.255358    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sn9hx"]
Nov 01 21:44:29.256877 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.256864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.259196 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.259172    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 21:44:29.259519 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.259502    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pth5p\""
Nov 01 21:44:29.260111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.260094    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 21:44:29.260716 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.260696    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 21:44:29.260823 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.260721    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266403    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjtfv\" (UniqueName: \"kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.266677 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.266575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gfv\" (UniqueName: \"kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.267114 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.267091    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bgzv8"]
Nov 01 21:44:29.269774 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.269752    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn9hx"]
Nov 01 21:44:29.273169 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.273146    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9jp45"]
Nov 01 21:44:29.274959 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.274945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.287441 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.287423    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jp45"]
Nov 01 21:44:29.359628 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359606    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 21:44:29.359763 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359714    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 21:44:29.359827 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359811    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 21:44:29.359872 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359859    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 21:44:29.359904 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359872    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pn6l\""
Nov 01 21:44:29.359904 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359859    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bzsh4\""
Nov 01 21:44:29.359970 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.359906    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 21:44:29.366996 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.366978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.367061 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367061 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.367138 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.367138 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.367204 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367204 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367268 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367209    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367268 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gfv\" (UniqueName: \"kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.367358 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xm7\" (UniqueName: \"kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.367395 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjtfv\" (UniqueName: \"kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.367642 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.367619    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.368029 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.368007    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.370998 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.370879    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.371071 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.370918    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.378786 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.378768    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjtfv\" (UniqueName: \"kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.379004 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.378984    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gfv\" (UniqueName: \"kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.468081 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xm7\" (UniqueName: \"kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.468171 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.468171 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.468268 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.468500 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468481    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.468689 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.468673    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.470050 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.470031    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.479506 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.479487    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xm7\" (UniqueName: \"kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.569627 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.569607    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:29.572770 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.572755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:29.584283 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.584264    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:29.859529 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.859443    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sn9hx"]
Nov 01 21:44:29.861486 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.861463    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jp45"]
Nov 01 21:44:29.861601 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:29.861510    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bgzv8"]
Nov 01 21:44:29.863383 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:29.863355    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ddd68d_9250_4231_8b29_ee881e0faeae.slice/crio-34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577 WatchSource:0}: Error finding container 34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577: Status 404 returned error can't find the container with id 34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577
Nov 01 21:44:29.864210 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:29.864183    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55adb8b3_868b_4fe9_a2ee_a2de94739289.slice/crio-6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c WatchSource:0}: Error finding container 6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c: Status 404 returned error can't find the container with id 6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c
Nov 01 21:44:29.865033 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:29.864968    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d58a7a_ace5_47e8_9d39_6ffcf04804af.slice/crio-a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44 WatchSource:0}: Error finding container a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44: Status 404 returned error can't find the container with id a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44
Nov 01 21:44:30.557538 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.557504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:30.557745 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.557726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:30.558037 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.558016    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:30.561907 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.561862    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 21:44:30.562861 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.562845    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 21:44:30.563021 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.563000    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\""
Nov 01 21:44:30.563117 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.563023    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\""
Nov 01 21:44:30.563184 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.563121    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 21:44:30.563240 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.563204    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\""
Nov 01 21:44:30.563285 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.563240    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 21:44:30.785364 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.785333    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jp45" event={"ID":"55adb8b3-868b-4fe9-a2ee-a2de94739289","Type":"ContainerStarted","Data":"6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c"}
Nov 01 21:44:30.787411 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.787381    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn9hx" event={"ID":"a2ddd68d-9250-4231-8b29-ee881e0faeae","Type":"ContainerStarted","Data":"34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577"}
Nov 01 21:44:30.790272 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.790246    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgzv8" event={"ID":"e7d58a7a-ace5-47e8-9d39-6ffcf04804af","Type":"ContainerStarted","Data":"8214fc528f32557a8cebe74f72d3c9c68517eb30351f35d01477c672a82407c7"}
Nov 01 21:44:30.790418 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.790312    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgzv8" event={"ID":"e7d58a7a-ace5-47e8-9d39-6ffcf04804af","Type":"ContainerStarted","Data":"b57b068eb619ff80e855dd4c1d5b73dba26c980b8d890176f6a9447829391a72"}
Nov 01 21:44:30.790418 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:30.790341    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgzv8" event={"ID":"e7d58a7a-ace5-47e8-9d39-6ffcf04804af","Type":"ContainerStarted","Data":"a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44"}
Nov 01 21:44:32.410214 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.410185    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"]
Nov 01 21:44:32.412413 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.412392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.422151 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.422085    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 21:44:32.422151 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.422077    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 21:44:32.423152 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.423134    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nrwmx\""
Nov 01 21:44:32.429158 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.429137    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 21:44:32.429356 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.429336    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 21:44:32.429444 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.429390    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 21:44:32.435019 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.434995    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"]
Nov 01 21:44:32.435118 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.435106    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"]
Nov 01 21:44:32.438114 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.438094    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hmnzz"]
Nov 01 21:44:32.438215 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.438184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.440187 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.440159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.446371 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.446352    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"]
Nov 01 21:44:32.451743 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.451726    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 21:44:32.451849 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.451801    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 21:44:32.451919 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.451908    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 21:44:32.452021 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.452007    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xhnrl\""
Nov 01 21:44:32.452343 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.452323    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 21:44:32.452439 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.452404    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bpkb5\""
Nov 01 21:44:32.491503 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.491607 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.491607 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.491728 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.491789 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.491874 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.491928 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c9q\" (UniqueName: \"kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.491976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.492016 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.491984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.492068 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.492120 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.492168 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.492208 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.492320 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.492441 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.492441 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.492527 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492462    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.492527 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jg2\" (UniqueName: \"kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.492527 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.492512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rws\" (UniqueName: \"kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.522834 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.522814    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 21:44:32.522976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.522958    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 21:44:32.593776 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c9q\" (UniqueName: \"kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.593939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.593994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jg2\" (UniqueName: \"kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rws\" (UniqueName: \"kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594218 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.594690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.594690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.594582    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.595236 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.595210    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.595669 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.595381    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.595669 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.595483    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.596090 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.596064    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.596090 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.596080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.596451 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.596417    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.596547 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.596511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.596611 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.596549    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.596611 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.595493    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.597680 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.597636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.597746 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.597695    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.597982 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.597960    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.598199 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.598172    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.599608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.599581    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.601225 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.601185    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.610834 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.610791    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rws\" (UniqueName: \"kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.611530 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.611502    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c9q\" (UniqueName: \"kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.611530 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.611517    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jg2\" (UniqueName: \"kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.726796 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.726769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:32.754999 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.754963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:32.762494 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.762470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:32.772800 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:32.772767    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c6f668_3618_4d04_9942_eab2dd9e6224.slice/crio-3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56 WatchSource:0}: Error finding container 3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56: Status 404 returned error can't find the container with id 3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56
Nov 01 21:44:32.800790 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.800763    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bgzv8" event={"ID":"e7d58a7a-ace5-47e8-9d39-6ffcf04804af","Type":"ContainerStarted","Data":"3c20d3bd19f2195f3560985502e10b536167707c6e0c8052bb96567f2e4b205e"}
Nov 01 21:44:32.805326 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.803188    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jp45" event={"ID":"55adb8b3-868b-4fe9-a2ee-a2de94739289","Type":"ContainerStarted","Data":"a8f2c017d8a9b794ced629f43ab5daf3e8a7d95819bef37fe72b76657a25788f"}
Nov 01 21:44:32.805326 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.803228    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jp45" event={"ID":"55adb8b3-868b-4fe9-a2ee-a2de94739289","Type":"ContainerStarted","Data":"e814ab8d58e1f3675f186ed89ba9a3f6e895040e05f11d8935388f56e7330fda"}
Nov 01 21:44:32.805326 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.803777    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:32.806422 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.806366    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmnzz" event={"ID":"13c6f668-3618-4d04-9942-eab2dd9e6224","Type":"ContainerStarted","Data":"3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56"}
Nov 01 21:44:32.846997 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.846947    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bgzv8" podStartSLOduration=1.676152227 podStartE2EDuration="3.846933108s" totalImagesPullingTime="2.170780881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:44:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:32.824270571 +0000 UTC m=+47.868368290" watchObservedRunningTime="2026-11-01 21:44:32.846933108 +0000 UTC m=+47.891030826"
Nov 01 21:44:32.847115 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:32.847086    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9jp45" podStartSLOduration=2.008302687 podStartE2EDuration="3.847077927s" totalImagesPullingTime="1.83877524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:32.846325877 +0000 UTC m=+47.890423597" watchObservedRunningTime="2026-11-01 21:44:32.847077927 +0000 UTC m=+47.891175647"
Nov 01 21:44:33.012236 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.012213    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"]
Nov 01 21:44:33.014791 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:33.014759    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba32e6c_796e_49bd_8042_db4de0b9e86b.slice/crio-4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7 WatchSource:0}: Error finding container 4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7: Status 404 returned error can't find the container with id 4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7
Nov 01 21:44:33.048760 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.048735    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"]
Nov 01 21:44:33.053268 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:33.053244    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5364a66c_4dcd_4819_a8a0_eb48a1b57f33.slice/crio-d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d WatchSource:0}: Error finding container d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d: Status 404 returned error can't find the container with id d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d
Nov 01 21:44:33.513611 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.513580    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:44:33.518269 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.517546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.521986 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521930    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 21:44:33.521986 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521950    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 21:44:33.521986 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521959    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 21:44:33.521986 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521962    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 21:44:33.522210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521995    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 21:44:33.522210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.522003    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 21:44:33.522210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.522007    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 21:44:33.522210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.521962    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 21:44:33.522210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.522069    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhrgg\""
Nov 01 21:44:33.522498 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.522481    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 21:44:33.534945 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.534917    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:44:33.602830 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.602793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.602985 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.602862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.602985 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.602947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.602982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603086 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603053    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603174 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603174 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603244 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603319 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603319 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603424 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603466 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603425    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.603507 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.603459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.704737 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.704699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.704920 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.704753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.704920 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.704791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.704920 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.704821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705144 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705112    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705286 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705408 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705408 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705520 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705520 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705520 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705662 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705662 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.705662 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.705597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.706783 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.706554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.706783 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.706691    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.707670 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.707622    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.709667 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.709641    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.709764 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.709725    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.709764 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.709757    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.709939 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.709916    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.710004 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.709978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.710149 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.710127    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.711624 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.711603    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.711842 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.711820    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.719762 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.719721    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.812596 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.812521    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9" event={"ID":"2ba32e6c-796e-49bd-8042-db4de0b9e86b","Type":"ContainerStarted","Data":"5b3c1aedf59306053592625e2a6c0fb7b294e13905a2fff3a5db37f820383fe3"}
Nov 01 21:44:33.812596 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.812586    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9" event={"ID":"2ba32e6c-796e-49bd-8042-db4de0b9e86b","Type":"ContainerStarted","Data":"bae44e578da9c752f2bab63e7fa373d89515693455ef89dc9e945b7e7c23dbb4"}
Nov 01 21:44:33.812813 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.812610    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9" event={"ID":"2ba32e6c-796e-49bd-8042-db4de0b9e86b","Type":"ContainerStarted","Data":"4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7"}
Nov 01 21:44:33.814682 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.814652    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sn9hx" event={"ID":"a2ddd68d-9250-4231-8b29-ee881e0faeae","Type":"ContainerStarted","Data":"05d5cbbde422c752e568014fa683f76503364feb68de1fc6e2d870ccfda13796"}
Nov 01 21:44:33.816819 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.816794    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn" event={"ID":"5364a66c-4dcd-4819-a8a0-eb48a1b57f33","Type":"ContainerStarted","Data":"d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d"}
Nov 01 21:44:33.834564 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.834535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:33.835395 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:33.835346    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sn9hx" podStartSLOduration=1.744347237 podStartE2EDuration="4.8353331s" totalImagesPullingTime="3.090985863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:33.834483513 +0000 UTC m=+48.878581233" watchObservedRunningTime="2026-11-01 21:44:33.8353331 +0000 UTC m=+48.879430821"
Nov 01 21:44:34.131650 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.131614    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:44:34.137206 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:34.137178    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf051ade6_cc37_418d_805c_673aaa131ae4.slice/crio-2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7 WatchSource:0}: Error finding container 2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7: Status 404 returned error can't find the container with id 2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7
Nov 01 21:44:34.507608 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.507583    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"]
Nov 01 21:44:34.510264 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.510236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.516430 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.516395    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 21:44:34.516810 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.516528    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 21:44:34.518564 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.518543    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5crl4\""
Nov 01 21:44:34.518807 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.518685    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 21:44:34.518807 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.518793    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 21:44:34.520606 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.519042    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-68sovd0l9bojt\""
Nov 01 21:44:34.520606 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.519229    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 21:44:34.524860 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.524840    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"]
Nov 01 21:44:34.614026 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614118 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614118 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614197 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdkj\" (UniqueName: \"kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614233 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614233 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614339 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614379 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.614409 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.614390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdkj\" (UniqueName: \"kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.716111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.715883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.717034 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.717013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.717394 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.717364    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.719649 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.719632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.721053 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.720442    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.721053 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.720950    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.721690 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.721645    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.721806 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.721780    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.722472 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.722429    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.731160 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.731138    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdkj\" (UniqueName: \"kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.821511 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.821480    2583 generic.go:356] "Generic (PLEG): container finished" podID="13c6f668-3618-4d04-9942-eab2dd9e6224" containerID="0f45da16f226dc92247939ef7e05945de84142b100d8bbe5b2710b0d7b55e02e" exitCode=0
Nov 01 21:44:34.821656 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.821587    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmnzz" event={"ID":"13c6f668-3618-4d04-9942-eab2dd9e6224","Type":"ContainerDied","Data":"0f45da16f226dc92247939ef7e05945de84142b100d8bbe5b2710b0d7b55e02e"}
Nov 01 21:44:34.824674 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.824644    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn" event={"ID":"5364a66c-4dcd-4819-a8a0-eb48a1b57f33","Type":"ContainerStarted","Data":"f516a9e8f3826dc691a1b087a6e76ccf09a24cd35f04d72102f7aedea0e4ae3f"}
Nov 01 21:44:34.824976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.824723    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn" event={"ID":"5364a66c-4dcd-4819-a8a0-eb48a1b57f33","Type":"ContainerStarted","Data":"df07ff0a0ba40a5767e006672e04fa414fe4384d2328ba229fbb3c6021680334"}
Nov 01 21:44:34.824976 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.824756    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn" event={"ID":"5364a66c-4dcd-4819-a8a0-eb48a1b57f33","Type":"ContainerStarted","Data":"815de0bd37c9e539b73b6fa150c874220b27480caebb8ff09458447e82f4cc78"}
Nov 01 21:44:34.826329 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.826252    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7"}
Nov 01 21:44:34.832591 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.832570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:34.832963 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.832942    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9" event={"ID":"2ba32e6c-796e-49bd-8042-db4de0b9e86b","Type":"ContainerStarted","Data":"4d683fcede73abcbe69f463709242d99f669c8f07ecd4d90fe5ebbc72ab5e4a4"}
Nov 01 21:44:34.868210 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:34.868157    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn" podStartSLOduration=1.357716084 podStartE2EDuration="2.868138111s" totalImagesPullingTime="1.510422027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:34.866754868 +0000 UTC m=+49.910852588" watchObservedRunningTime="2026-11-01 21:44:34.868138111 +0000 UTC m=+49.912235831"
Nov 01 21:44:35.124119 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.124060    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9" podStartSLOduration=1.8204025719999999 podStartE2EDuration="3.124045238s" totalImagesPullingTime="1.303642666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:34.893447999 +0000 UTC m=+49.937545735" watchObservedRunningTime="2026-11-01 21:44:35.124045238 +0000 UTC m=+50.168142956"
Nov 01 21:44:35.124883 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.124862    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"]
Nov 01 21:44:35.128128 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:35.128102    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf109cd89_d922_4952_b092_0360bbbbaf33.slice/crio-8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850 WatchSource:0}: Error finding container 8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850: Status 404 returned error can't find the container with id 8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850
Nov 01 21:44:35.838714 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.838673    2583 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="4894384708fc5d0fed581faf982955a8332ee38f62787ff73db2b4862eab0025" exitCode=0
Nov 01 21:44:35.839139 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.838758    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"4894384708fc5d0fed581faf982955a8332ee38f62787ff73db2b4862eab0025"}
Nov 01 21:44:35.840638 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.840565    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850"}
Nov 01 21:44:35.843621 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.843596    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmnzz" event={"ID":"13c6f668-3618-4d04-9942-eab2dd9e6224","Type":"ContainerStarted","Data":"07a6584cf3da20825f4b895d9cbd7bf7574b1a825c200b2a073e78936b2d4ab9"}
Nov 01 21:44:35.843713 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.843649    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hmnzz" event={"ID":"13c6f668-3618-4d04-9942-eab2dd9e6224","Type":"ContainerStarted","Data":"b5cf97cb50f318f1ad81105579f242d41296956b205bc9846cd6120fdcaffae8"}
Nov 01 21:44:35.893139 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:35.892341    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hmnzz" podStartSLOduration=2.674778379 podStartE2EDuration="3.892322741s" totalImagesPullingTime="1.135008602s" totalInitContainerRuntime="82.53576ms" isStatefulPod=true podCreationTimestamp="2026-11-01 21:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:35.89079225 +0000 UTC m=+50.934889981" watchObservedRunningTime="2026-11-01 21:44:35.892322741 +0000 UTC m=+50.936420462"
Nov 01 21:44:36.824656 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.824620    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f99686fb5-2wbs5"]
Nov 01 21:44:36.826899 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.826867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.835832 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.835805    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 21:44:36.836353 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.836333    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 21:44:36.841730 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.841707    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f99686fb5-2wbs5"]
Nov 01 21:44:36.851110 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.851060    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 21:44:36.851110 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.851070    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nk9sj\""
Nov 01 21:44:36.851110 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.851098    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 21:44:36.852125 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.852097    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-26v4dnni8p4ij\""
Nov 01 21:44:36.936716 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.936685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.936878 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.936749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.936956 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.936897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6rl\" (UniqueName: \"kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.937018 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.936956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.937171 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.937144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.937247 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.937190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:36.937247 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:36.937219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038276 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6rl\" (UniqueName: \"kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038464 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038654 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.038654 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.038511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.039722 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.039691    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.040670 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.040641    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.041251 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.041224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.043823 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.043801    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.044120 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.044094    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.051567 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.051536    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6rl\" (UniqueName: \"kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.056983 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.056963    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.140560 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.140534    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:37.242009 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.241218    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"]
Nov 01 21:44:37.244267 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.244246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.256255 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.256221    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"]
Nov 01 21:44:37.256965 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.256938    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 21:44:37.257054 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.256998    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-59b8p\""
Nov 01 21:44:37.340662 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.340633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.340795 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.340775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twjx\" (UniqueName: \"kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.424798 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.424656    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f99686fb5-2wbs5"]
Nov 01 21:44:37.427133 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:37.427106    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeec369_c249_494b_b2a2_8c0753b3d17e.slice/crio-9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973 WatchSource:0}: Error finding container 9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973: Status 404 returned error can't find the container with id 9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973
Nov 01 21:44:37.441358 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.441325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.441577 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.441547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twjx\" (UniqueName: \"kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.444039 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.444015    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.464881 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.464852    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twjx\" (UniqueName: \"kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.575914 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.575893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:37.850484 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.850420    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"]
Nov 01 21:44:37.852580 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.852552    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" event={"ID":"cfeec369-c249-494b-b2a2-8c0753b3d17e","Type":"ContainerStarted","Data":"9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973"}
Nov 01 21:44:37.853183 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:37.853162    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefe7ab7_6f8a_4895_af7f_747da1036866.slice/crio-9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9 WatchSource:0}: Error finding container 9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9: Status 404 returned error can't find the container with id 9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9
Nov 01 21:44:37.856683 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.856519    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"c1b79eec21aca3e12b9eac1ae07a74b182431bf9dd6590fc990158c3b29f3b7a"}
Nov 01 21:44:37.856683 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.856572    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"49d1775adc0b211c4f1eb6713e074144f812c4b976e9c34eb676ed602e151002"}
Nov 01 21:44:37.856683 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.856611    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"fee4bbeae741b09de7c1641db1d21a33554b9f36b028ec6ac84a832f04ebfc9d"}
Nov 01 21:44:37.856683 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.856639    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"14bb91ca455545f819c89a223b08e68dfa01732644e524f51f42fa844cd0b30c"}
Nov 01 21:44:37.858868 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.858848    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"333619a09fc47a5d511097941d87b0cefac9194fa70763c7eaa7d2abd86ef1a8"}
Nov 01 21:44:37.858933 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.858897    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"eb79cd2a4b380935374ff16673a10e247664db328a7a0ed4a4d828baafc1a66d"}
Nov 01 21:44:37.858933 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:37.858923    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"8af0d660bac13b4425b8f7efee5b444eff8cbbb539bbadbeab92f75ba7c8a97b"}
Nov 01 21:44:38.759503 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.759474    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:44:38.762272 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.762251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.771897 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.771873    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 21:44:38.772010 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.771892    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 21:44:38.772010 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.771879    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xhnwq\""
Nov 01 21:44:38.772010 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.771993    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 21:44:38.772175 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.772032    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 21:44:38.772175 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.772045    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 21:44:38.772175 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.772098    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 21:44:38.772175 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.772137    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 21:44:38.777984 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.777953    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 21:44:38.779450 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.779425    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 21:44:38.779450 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.779440    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 21:44:38.779594 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.779471    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 21:44:38.779594 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.779569    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-chkoj1b7amdmk\""
Nov 01 21:44:38.779723 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.779705    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 21:44:38.780586 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.780566    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 21:44:38.780586 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.780579    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 21:44:38.784779 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.784758    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:44:38.859412 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859679    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.859913 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.859996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.860239 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.860188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.869460 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.869436    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"bc6b0ac69c6d55b2ff21f397f4db1eba019f29860fa2cb822c6a7056b93b1807"}
Nov 01 21:44:38.869564 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.869534    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerStarted","Data":"fa85cc7366974bca56ad37f90cef4358840931f34f82404cafcb7f78a8b130c4"}
Nov 01 21:44:38.872627 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.872607    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"5175de5bdf407a3c91e43d4d5335ad750c2ae9adb51f77bcf215c738644f075c"}
Nov 01 21:44:38.872711 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.872658    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"68a2bea56723e69f17b0123e5d26ba9a26e2a4b0b20655ef4a48396254c211b3"}
Nov 01 21:44:38.872711 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.872685    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" event={"ID":"f109cd89-d922-4952-b092-0360bbbbaf33","Type":"ContainerStarted","Data":"023f43dbf42422a94cf0ce67e72542933337f8af850c20a9cee6da0659985413"}
Nov 01 21:44:38.872791 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.872723    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:38.874082 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.874061    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" event={"ID":"cefe7ab7-6f8a-4895-af7f-747da1036866","Type":"ContainerStarted","Data":"9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9"}
Nov 01 21:44:38.905126 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.905041    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.554683088 podStartE2EDuration="5.905028211s" totalImagesPullingTime="3.337393213s" totalInitContainerRuntime="12.95191ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:38.903056812 +0000 UTC m=+53.947154541" watchObservedRunningTime="2026-11-01 21:44:38.905028211 +0000 UTC m=+53.949125930"
Nov 01 21:44:38.929945 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.929907    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podStartSLOduration=2.033382592 podStartE2EDuration="4.929892899s" totalImagesPullingTime="2.896510307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:38.928810423 +0000 UTC m=+53.972908141" watchObservedRunningTime="2026-11-01 21:44:38.929892899 +0000 UTC m=+53.973990651"
Nov 01 21:44:38.961104 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961249 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961249 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961249 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961621 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961748 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961748 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961858 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961858 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961858 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.961858 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961947    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962051 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.961999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.962060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.962114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.962173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962362 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.962270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.962567 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.962368    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.963060 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.963035    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.964027 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.964000    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.964461    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965121    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965256    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965593    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965684    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965759    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966030 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.965947    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.966736 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.966711    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.967285 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.967262    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.967789 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.967764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.969150 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.969108    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.969710 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.969687    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.970084 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.970047    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.970308 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.970274    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.970533 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.970510    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:38.980203 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:38.980182    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:39.084281 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.084211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:39.385594 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.385442    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:44:39.388140 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:39.388113    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb06ecb_8200_42eb_979e_d407d18b8f10.slice/crio-1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550 WatchSource:0}: Error finding container 1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550: Status 404 returned error can't find the container with id 1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550
Nov 01 21:44:39.878743 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.878703    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" event={"ID":"cfeec369-c249-494b-b2a2-8c0753b3d17e","Type":"ContainerStarted","Data":"5e252301f4bb73c5f044f884e505e8b02bd5de82b66c1dbb906e59c314e11ae4"}
Nov 01 21:44:39.880111 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.880088    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" event={"ID":"cefe7ab7-6f8a-4895-af7f-747da1036866","Type":"ContainerStarted","Data":"72104668d4bc97c9ae54ba663efd63ec0c22ea3e3705856a1a9f30c18e688d0e"}
Nov 01 21:44:39.880249 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.880228    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:39.881816 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.881794    2583 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1" exitCode=0
Nov 01 21:44:39.881916 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.881854    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"}
Nov 01 21:44:39.881974 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.881914    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550"}
Nov 01 21:44:39.885694 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.885675    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:39.902597 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.902530    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podStartSLOduration=2.471409791 podStartE2EDuration="3.902519559s" totalImagesPullingTime="1.431109768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:39.901412828 +0000 UTC m=+54.945510547" watchObservedRunningTime="2026-11-01 21:44:39.902519559 +0000 UTC m=+54.946617278"
Nov 01 21:44:39.959308 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:39.959245    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podStartSLOduration=1.217634686 podStartE2EDuration="2.95923319s" totalImagesPullingTime="1.741598504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:39.958184256 +0000 UTC m=+55.002281978" watchObservedRunningTime="2026-11-01 21:44:39.95923319 +0000 UTC m=+55.003330908"
Nov 01 21:44:43.835547 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.835516    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:43.905329 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905270    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"}
Nov 01 21:44:43.905469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905361    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"}
Nov 01 21:44:43.905469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905404    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"}
Nov 01 21:44:43.905469 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905443    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"}
Nov 01 21:44:43.905561 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905485    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"}
Nov 01 21:44:43.905561 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.905520    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerStarted","Data":"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"}
Nov 01 21:44:43.941499 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:43.941436    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.841957464 podStartE2EDuration="5.941421362s" totalImagesPullingTime="3.078174316s" totalInitContainerRuntime="21.289582ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:43.940713425 +0000 UTC m=+58.984811155" watchObservedRunningTime="2026-11-01 21:44:43.941421362 +0000 UTC m=+58.985519080"
Nov 01 21:44:44.084344 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:44.084289    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:44.888482 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:44.888457    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:47.767219 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:47.767191    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:51.168287 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.168245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:51.171248 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.171231    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 21:44:51.181530 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.181506    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:51.268979 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.268952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:51.269101 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.269003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:51.271753 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.271734    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 21:44:51.271955 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.271939    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 21:44:51.281125 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.281103    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:51.281621 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.281606    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 21:44:51.281676 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.281666    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\""
Nov 01 21:44:51.289894 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.289878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:51.291998 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.291977    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:51.562899 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.562879    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bcxfp"]
Nov 01 21:44:51.565051 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:51.565024    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedd4f98_22d8_4bbf_8e7a_6c54d1504e1b.slice/crio-08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde WatchSource:0}: Error finding container 08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde: Status 404 returned error can't find the container with id 08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde
Nov 01 21:44:51.574401 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.574383    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\""
Nov 01 21:44:51.582451 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.582432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:51.585152 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.585135    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\""
Nov 01 21:44:51.593434 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.593414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:51.859825 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.859800    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f4s8z"]
Nov 01 21:44:51.862037 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:51.862013    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2851cfd_c9b5_40db_8ddb_93ad5ba9fb61.slice/crio-890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6 WatchSource:0}: Error finding container 890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6: Status 404 returned error can't find the container with id 890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6
Nov 01 21:44:51.878354 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.878234    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5s44"]
Nov 01 21:44:51.880635 ip-10-0-135-146 kubenswrapper[2583]: W1101 21:44:51.880606    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03eefce5_3634_456f_889c_3ea91faaff0d.slice/crio-d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059 WatchSource:0}: Error finding container d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059: Status 404 returned error can't find the container with id d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059
Nov 01 21:44:51.934205 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.934179    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcxfp" event={"ID":"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b","Type":"ContainerStarted","Data":"08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde"}
Nov 01 21:44:51.935360 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.935334    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f4s8z" event={"ID":"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61","Type":"ContainerStarted","Data":"890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6"}
Nov 01 21:44:51.936332 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:51.936312    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5s44" event={"ID":"03eefce5-3634-456f-889c-3ea91faaff0d","Type":"ContainerStarted","Data":"d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059"}
Nov 01 21:44:52.945117 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:52.945025    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcxfp" event={"ID":"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b","Type":"ContainerStarted","Data":"fc7036e38c5070733eb3bf05ebd53fa00b985c3ae043b7a6bf59008c62a35a2a"}
Nov 01 21:44:52.945619 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:52.945132    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcxfp" event={"ID":"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b","Type":"ContainerStarted","Data":"c041031cbb20d73eb2d9a96f9a800537c0fc073bb9d5d494c1a06b488ea4f5e1"}
Nov 01 21:44:52.966317 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:52.966246    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bcxfp" podStartSLOduration=66.936586261 podStartE2EDuration="1m7.966231575s" totalImagesPullingTime="1.029645314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:52.964261607 +0000 UTC m=+68.008359330" watchObservedRunningTime="2026-11-01 21:44:52.966231575 +0000 UTC m=+68.010329296"
Nov 01 21:44:55.960262 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:55.960220    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5s44" event={"ID":"03eefce5-3634-456f-889c-3ea91faaff0d","Type":"ContainerStarted","Data":"ec86fe13f4b0bde298ccb92b07cab199cd2d402535b53ffaf4d5926a8f475394"}
Nov 01 21:44:55.960705 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:55.960364    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:55.978314 ip-10-0-135-146 kubenswrapper[2583]: I1101 21:44:55.978240    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f5s44" podStartSLOduration=67.672417587 podStartE2EDuration="1m10.978222868s" totalImagesPullingTime="3.305805281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:44:55.977450267 +0000 UTC m=+71.021547987" watchObservedRunningTime="2026-11-01 21:44:55.978222868 +0000 UTC m=+71.022320588"
Nov 01 21:44:56.686233 ip-10-0-135-146 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 21:44:56.699727 ip-10-0-135-146 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 21:44:56.699911 ip-10-0-135-146 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 21:44:56.700121 ip-10-0-135-146 systemd[1]: kubelet.service: Consumed 4.280s CPU time.
Nov 01 21:44:56.712764 ip-10-0-135-146 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 21:44:56.775313 ip-10-0-135-146 kubenswrapper[5845]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:44:56.775313 ip-10-0-135-146 kubenswrapper[5845]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 21:44:56.775313 ip-10-0-135-146 kubenswrapper[5845]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:44:56.775313 ip-10-0-135-146 kubenswrapper[5845]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 21:44:56.777286 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777214    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.777286 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777287    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777291    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777311    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777317    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777321    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777326    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777330    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777334    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777338    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777341    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777345    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777349    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777353    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777357    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777361    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777364    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777368    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777374    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777381    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.777392 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777387    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777391    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777395    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777399    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777402    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777405    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777407    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777418    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777421    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777424    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777426    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777429    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777432    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777434    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777437    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777440    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777442    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777445    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777448    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777450    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.777918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777453    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777455    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777458    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777460    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777463    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777465    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777469    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777471    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777474    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777476    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777479    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777482    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777484    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777487    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777491    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777494    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777498    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777501    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777504    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.778425 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777506    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777509    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777511    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777514    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777517    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777520    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777522    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777525    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777528    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777531    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777533    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777536    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777538    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777541    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777544    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777546    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777549    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777551    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777554    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777556    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.778904 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777560    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777563    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777565    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777567    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777570    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777573    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777575    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777579    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777583    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777587    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777590    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777593    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777597    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777601    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777604    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777607    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777610    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777613    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777616    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.779457 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777619    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777621    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777624    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777627    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777629    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777632    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777634    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.777637    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778513    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778522    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778526    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778529    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778532    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778534    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778543    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778546    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778549    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778551    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778555    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778558    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.779936 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778561    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778564    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778566    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778569    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778571    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778574    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778577    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778579    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778582    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778584    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778587    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778590    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778592    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778595    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778597    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778600    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778602    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778605    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778608    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778612    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.780465 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778614    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778617    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778620    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778622    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778627    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778631    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778634    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778642    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778645    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778648    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778650    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778653    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778656    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778659    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778662    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778665    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778667    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778670    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778672    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.780958 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778675    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778678    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778681    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778683    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778686    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778688    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778691    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778695    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778697    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778700    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778703    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778705    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778708    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778710    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778713    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778716    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778718    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778721    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778723    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.781440 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778726    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778728    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778740    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778743    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778745    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778748    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778751    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778753    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778756    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778759    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778761    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778764    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778767    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778770    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778772    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778775    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778777    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778780    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778783    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778791    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.781909 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778795    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778797    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778800    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778802    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778805    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778807    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778810    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778813    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778815    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778818    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778820    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778823    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778825    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778828    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778831    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.778843    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779401    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779407    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779413    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.782431 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779417    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779419    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779422    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779425    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779428    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779431    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779433    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779436    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779438    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779441    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779443    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779446    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779448    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779451    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779453    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779456    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779458    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779461    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779464    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779466    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.782896 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779469    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779472    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779474    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779477    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779480    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779482    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779485    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779488    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779491    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779494    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779497    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779501    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779504    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779507    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779509    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779512    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779514    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779517    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779519    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779521    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.783485 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779524    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779527    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779529    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779532    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779534    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779537    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779539    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779542    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779544    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779547    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779549    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779552    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779560    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779565    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779568    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779570    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779574    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779576    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779579    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.783994 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779581    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779584    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779587    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779590    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779593    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779596    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779598    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779601    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779604    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779606    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779609    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779611    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779614    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779617    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779620    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779622    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779624    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779627    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779629    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779632    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.784484 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779635    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779639    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779642    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779646    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779650    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779653    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779656    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779659    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779661    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779664    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779667    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779669    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779672    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779674    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779677    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779680    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779684    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779686    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779689    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.785071 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779691    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779695    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779697    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779700    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.779702    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779799    5845 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779806    5845 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779818    5845 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779823    5845 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779828    5845 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779831    5845 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779835    5845 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779839    5845 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779842    5845 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779845    5845 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779849    5845 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779853    5845 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779856    5845 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779858    5845 flags.go:64] FLAG: --cgroup-root=""
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779861    5845 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779864    5845 flags.go:64] FLAG: --client-ca-file=""
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779867    5845 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779870    5845 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779875    5845 flags.go:64] FLAG: --cluster-domain=""
Nov 01 21:44:56.785837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779878    5845 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779881    5845 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779884    5845 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779887    5845 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779891    5845 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779896    5845 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779899    5845 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779903    5845 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779906    5845 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779910    5845 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779913    5845 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779916    5845 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779920    5845 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779924    5845 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779927    5845 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779930    5845 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779933    5845 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779936    5845 flags.go:64] FLAG: --enable-server="true"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779939    5845 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779943    5845 flags.go:64] FLAG: --event-burst="100"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779946    5845 flags.go:64] FLAG: --event-qps="50"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779949    5845 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779953    5845 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779956    5845 flags.go:64] FLAG: --eviction-hard=""
Nov 01 21:44:56.786600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779960    5845 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779963    5845 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779966    5845 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779969    5845 flags.go:64] FLAG: --eviction-soft=""
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779972    5845 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779975    5845 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779978    5845 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779981    5845 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779984    5845 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779987    5845 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779989    5845 flags.go:64] FLAG: --feature-gates=""
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779994    5845 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.779997    5845 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780001    5845 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780004    5845 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780007    5845 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780011    5845 flags.go:64] FLAG: --help="false"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780017    5845 flags.go:64] FLAG: --hostname-override="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780020    5845 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780023    5845 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780026    5845 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780030    5845 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780033    5845 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 21:44:56.787194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780036    5845 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780039    5845 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780042    5845 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780045    5845 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780048    5845 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780051    5845 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780054    5845 flags.go:64] FLAG: --kube-reserved=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780057    5845 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780060    5845 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780063    5845 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780066    5845 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780069    5845 flags.go:64] FLAG: --lock-file=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780072    5845 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780075    5845 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780078    5845 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780083    5845 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780086    5845 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780089    5845 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780092    5845 flags.go:64] FLAG: --logging-format="text"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780095    5845 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780098    5845 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780101    5845 flags.go:64] FLAG: --manifest-url=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780104    5845 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780108    5845 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780112    5845 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 21:44:56.787784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780116    5845 flags.go:64] FLAG: --max-pods="110"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780120    5845 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780124    5845 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780127    5845 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780130    5845 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780133    5845 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780136    5845 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780139    5845 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780146    5845 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780149    5845 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780152    5845 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780155    5845 flags.go:64] FLAG: --pod-cidr=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780158    5845 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780161    5845 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780164    5845 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780167    5845 flags.go:64] FLAG: --port="10250"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780170    5845 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780174    5845 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-05431e098173a1750"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780177    5845 flags.go:64] FLAG: --qos-reserved=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780180    5845 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780183    5845 flags.go:64] FLAG: --register-node="true"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780186    5845 flags.go:64] FLAG: --register-with-taints=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780193    5845 flags.go:64] FLAG: --registry-burst="10"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780196    5845 flags.go:64] FLAG: --registry-qps="5"
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780199    5845 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780202    5845 flags.go:64] FLAG: --reserved-memory=""
Nov 01 21:44:56.788494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780206    5845 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780209    5845 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780212    5845 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780215    5845 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780218    5845 flags.go:64] FLAG: --runonce="false"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780221    5845 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780224    5845 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780229    5845 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780232    5845 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780235    5845 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780238    5845 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780241    5845 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780244    5845 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780247    5845 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780250    5845 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780253    5845 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780256    5845 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780259    5845 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780262    5845 flags.go:64] FLAG: --system-cgroups=""
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780265    5845 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780270    5845 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780273    5845 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780276    5845 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780281    5845 flags.go:64] FLAG: --tls-min-version=""
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780284    5845 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 21:44:56.789182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780287    5845 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780289    5845 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780307    5845 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780311    5845 flags.go:64] FLAG: --v="2"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780316    5845 flags.go:64] FLAG: --version="false"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780320    5845 flags.go:64] FLAG: --vmodule=""
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780324    5845 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780327    5845 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780474    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780479    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780482    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780485    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780488    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780491    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780496    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780499    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780502    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780504    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780507    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780510    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780513    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.789890 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780516    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780518    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780521    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780523    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780526    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780529    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780533    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780536    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780538    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780541    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780544    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780546    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780549    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780551    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780554    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780556    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780559    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780562    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780566    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.790569 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780570    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780573    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780576    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780578    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780581    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780583    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780587    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780593    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780596    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780598    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780601    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780603    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780606    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780609    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780611    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780614    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780616    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780619    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780622    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.791052 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780626    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780629    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780631    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780634    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780637    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780639    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780642    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780644    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780647    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780649    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780652    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780654    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780657    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780659    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780662    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780664    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780667    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780670    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780672    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780676    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.791553 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780679    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780682    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780684    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780687    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780689    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780692    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780695    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780697    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780700    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780702    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780705    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780707    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780711    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780714    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780716    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780719    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780722    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780724    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780727    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.792145 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780730    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780732    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780735    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780737    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780740    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780743    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780745    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780748    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780750    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780753    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780755    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780758    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780763    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780767    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780769    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780772    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.792622 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.780797    5845 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780833    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780837    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780840    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780844    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780847    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780849    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780852    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780855    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780859    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780862    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780864    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780867    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780870    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780872    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780875    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780878    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780880    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780883    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780885    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.793019 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780888    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780891    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780893    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780896    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780900    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780903    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780906    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780909    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780911    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780915    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780918    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780921    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780924    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780926    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780929    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780931    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780934    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780937    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780939    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.793878 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780942    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780944    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780948    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780951    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780954    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780956    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780959    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780962    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780965    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780967    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780970    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780973    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780976    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780978    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780980    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780984    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780986    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780989    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780991    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780994    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.794430 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780997    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.780999    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781003    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781006    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781008    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781011    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781014    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781016    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781019    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781022    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781024    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781027    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781030    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781032    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781036    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781039    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781042    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781044    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781047    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.794918 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781049    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781052    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781054    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781057    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781060    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781062    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781065    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781067    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781070    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781072    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781075    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781077    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781080    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781083    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781085    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781089    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781092    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781095    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781098    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781101    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.795438 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781104    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781107    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781111    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781113    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781116    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781118    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781121    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781124    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.781127    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.785613    5845 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.785653    5845 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.785702    5845 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.785710    5845 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786002    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786012    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786017    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786020    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786025    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786028    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.795934 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786030    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786033    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786036    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786039    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786042    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786044    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786047    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786050    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786053    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786055    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786058    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786061    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786064    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786066    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786069    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786072    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786074    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786077    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786079    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.796486 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786082    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786086    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786088    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786091    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786094    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786096    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786099    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786102    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786105    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786107    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786110    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786115    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786117    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786120    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786122    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786125    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786129    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786131    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786134    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786136    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.796981 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786139    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786141    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786144    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786147    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786149    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786152    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786155    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786158    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786160    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786165    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786169    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786173    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786176    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786179    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786182    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786184    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786188    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786191    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786193    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.797477 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786196    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786198    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786201    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786204    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786206    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786209    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786211    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786214    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786216    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786219    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786222    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786224    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786227    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786229    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786232    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786236    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786239    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786241    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786245    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.797947 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786248    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786250    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786253    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786256    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786259    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786262    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786264    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786267    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786269    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786272    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786275    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786277    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786281    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786284    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786288    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786291    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786307    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786310    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786313    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786316    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.798421 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786318    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786321    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786324    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.786354    5845 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786493    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786498    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786501    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786504    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786506    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786510    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786513    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786516    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786518    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786521    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786523    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786526    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.798898 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786529    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786532    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786534    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786537    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786540    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786543    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786545    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786548    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786551    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786554    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786556    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786559    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786561    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786564    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786568    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786572    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786575    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786578    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786581    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.799280 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786584    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786586    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786589    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786592    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786594    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786597    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786600    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786603    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786607    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786609    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786612    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786615    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786618    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786621    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786623    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786626    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786628    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786631    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786633    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.799771 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786636    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786638    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786641    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786643    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786646    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786649    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786651    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786654    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786656    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786659    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786661    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786664    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786666    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786669    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786671    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786674    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786676    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786679    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786681    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.800222 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786684    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786686    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786689    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786692    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786695    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786697    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786700    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786702    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786705    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786707    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786710    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786712    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786715    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786717    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786720    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786723    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786725    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786727    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786730    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786733    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.800773 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786735    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786738    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786741    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786743    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786746    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786748    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786751    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786754    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786756    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786759    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786761    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786764    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786766    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786769    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786773    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786777    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.801574 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786780    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.786803    5845 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786832    5845 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786837    5845 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786840    5845 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786844    5845 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786847    5845 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786850    5845 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786853    5845 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786856    5845 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786858    5845 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786861    5845 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786863    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786866    5845 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786868    5845 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786871    5845 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 21:44:56.802129 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786874    5845 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786876    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786879    5845 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786881    5845 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786884    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786886    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786889    5845 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786891    5845 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786894    5845 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786896    5845 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786899    5845 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786901    5845 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786904    5845 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786907    5845 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786912    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786915    5845 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786918    5845 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786921    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786923    5845 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786926    5845 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 21:44:56.802803 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786929    5845 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786931    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786934    5845 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786936    5845 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786939    5845 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786941    5845 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786944    5845 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786946    5845 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786949    5845 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786951    5845 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786954    5845 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786956    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786959    5845 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786962    5845 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786964    5845 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786967    5845 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786970    5845 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786972    5845 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786975    5845 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786977    5845 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 21:44:56.803624 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786980    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786982    5845 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786986    5845 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786991    5845 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786995    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.786998    5845 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787003    5845 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787006    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787009    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787012    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787016    5845 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787019    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787021    5845 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787024    5845 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787027    5845 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787029    5845 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787032    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787035    5845 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787037    5845 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 21:44:56.804112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787040    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787042    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787045    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787048    5845 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787050    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787053    5845 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787055    5845 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787058    5845 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787060    5845 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787063    5845 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787065    5845 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787068    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787070    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787073    5845 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787075    5845 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787078    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787082    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787084    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787087    5845 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 21:44:56.804616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787091    5845 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787093    5845 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787097    5845 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787100    5845 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787103    5845 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787105    5845 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787108    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787110    5845 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787113    5845 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787116    5845 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787118    5845 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787121    5845 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787123    5845 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:44:56.787126    5845 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.787173    5845 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788097    5845 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788145    5845 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788327    5845 server.go:1014] "Starting client certificate rotation"
Nov 01 21:44:56.805112 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788455    5845 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788500    5845 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 21:38:45 +0000 UTC" deadline="2028-04-03 10:56:56.504544427 +0000 UTC"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788534    5845 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12445h11m59.716012552s"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.788742    5845 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.789757    5845 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.791233    5845 log.go:25] "Validated CRI v1 runtime API"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.793219    5845 log.go:25] "Validated CRI v1 image API"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.793914    5845 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 21:44:56.805568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.796036    5845 fs.go:135] Filesystem UUIDs: map[2e88aefb-6dad-4cdb-8af9-d5839ee96cb7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c24973aa-6c3b-4f8b-9211-63d0bac5a9ca:/dev/nvme0n1p4]
Nov 01 21:44:56.805794 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.796050    5845 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde/userdata/shm major:0 minor:476 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550/userdata/shm major:0 minor:432 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059/userdata/shm major:0 minor:482 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03eefce5-3634-456f-889c-3ea91faaff0d/volumes/kubernetes.io~projected/kube-api-access-4zqlt:{mountpoint:/var/lib/kubelet/pods/03eefce5-3634-456f-889c-3ea91faaff0d/volumes/kubernetes.io~projected/kube-api-access-4zqlt major:0 minor:475 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~projected/kube-api-access-n2rws:{mountpoint:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~projected/kube-api-access-n2rws major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~projected/kube-api-access-z9c9q:{mountpoint:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~projected/kube-api-access-z9c9q major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~projected/kube-api-access-tv58b:{mountpoint:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~projected/kube-api-access-tv58b major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40b8c97d-537b-46e4-aa07-339c0d0b225b/volumes/kubernetes.io~projected/kube-api-access-t57cl:{mountpoint:/var/lib/kubelet/pods/40b8c97d-537b-46e4-aa07-339c0d0b225b/volumes/kubernetes.io~projected/kube-api-access-t57cl major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43d867fa-5c9f-4cab-ae4e-08f4e3296a81/volumes/kubernetes.io~projected/kube-api-access-976g5:{mountpoint:/var/lib/kubelet/pods/43d867fa-5c9f-4cab-ae4e-08f4e3296a81/volumes/kubernetes.io~projected/kube-api-access-976g5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~projected/kube-api-access-z9jg2:{mountpoint:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~projected/kube-api-access-z9jg2 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~projected/kube-api-access-f9xm7:{mountpoint:/var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~projected/kube-api-access-f9xm7 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d602079-f8c5-49f7-afe1-8d6e074b8c27/volumes/kubernetes.io~projected/kube-api-access-hhc56:{mountpoint:/var/lib/kubelet/pods/8d602079-f8c5-49f7-afe1-8d6e074b8c27/volumes/kubernetes.io~projected/kube-api-access-hhc56 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98dd454d-3bb0-48d9-b6ed-d907da241372/volumes/kubernetes.io~projected/kube-api-access-f9n7x:{mountpoint:/var/lib/kubelet/pods/98dd454d-3bb0-48d9-b6ed-d907da241372/volumes/kubernetes.io~projected/kube-api-access-f9n7x major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~projected/kube-api-access-78gfv:{mountpoint:/var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~projected/kube-api-access-78gfv major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf3bf5f-cbda-43b8-b263-dad640f9f5a3/volumes/kubernetes.io~projected/kube-api-access-cdfrk:{mountpoint:/var/lib/kubelet/pods/baf3bf5f-cbda-43b8-b263-dad640f9f5a3/volumes/kubernetes.io~projected/kube-api-access-cdfrk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~projected/kube-api-access-wmp4t:{mountpoint:/var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~projected/kube-api-access-wmp4t major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~projected/kube-api-access-7twjx:{mountpoint:/var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~projected/kube-api-access-7twjx major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~projected/kube-api-access-9g6rl:{mountpoint:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~projected/kube-api-access-9g6rl major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dac098ba-33a8-4a99-812f-ae5561684d7c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dac098ba-33a8-4a99-812f-ae5561684d7c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/config-reloader/2 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/prometheus/15 major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~empty-dir/config-out major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/kube-api-access-2b67s:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/kube-api-access-2b67s major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/tls-assets major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/config major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/web-config major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~projected/kube-api-access-rjtfv:{mountpoint:/var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~projected/kube-api-access-rjtfv major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~projected/kube-api-access-mr5kx:{mountpoint:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~projected/kube-api-access-mr5kx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/alertmanager/9 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/kube-api-access-z7ctf:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/kube-api-access-z7ctf major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/tls-assets major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/config-volume major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/web-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0aef984-8fc9-46e0-8243-27b9935991a3/volumes/kubernetes.io~projected/kube-api-access-746xf:{mountpoint:/var/lib/kubelet/pods/f0aef984-8fc9-46e0-8243-27b9935991a3/volumes/kubernetes.io~projected/kube-api-access-746xf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~projected/kube-api-access-6xdkj:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~projected/kube-api-access-6xdkj major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:320 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a5d92424ca065187e9dd8c15ea2635664711faf7729ec8bd73da2acc509ea4fa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0e92c147decbd0bf406614c1306e5fb2fdea6f4f666466a6d050f445daa7ef62/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b310089e89d490df56de4e7c52fa3b745a2f82f4fb95c80d42c953be9649eb27/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/e92aa4bdf8cab184ba574d0b9d8b8601029d06f020fcd46df602e71bf32a52ea/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/8d24345d3de02b73ffb7afb115d6a5c0a302dbaec2e81f494105f6581bde20d8/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3c884458cd7165eb25495024759e0d4d5579818171efcf8ec98fb648278742ee/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/82b93e600978be2dfd09fcfcdab718012cc005b4d08f9143068e02c3225980a3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1adb80fbcf80695bd080d0e6b46404dfb150fa3c48a11f5b54b83f514c56600e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/275dcded7140d83fc82a99560989b4dde00df41a3208ae41d0cade58b01f7c83/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2e7603a0b15fd045e8a77c13f8d18784525bc2aecd25825e12f66c2db57f2164/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/9c0613bf45986f18779c4da319bd091f32b2c3e8b7827069840ba0959372478d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/fbe65505fdf13480ba9176f52a598c6eca2175f3c3516a2e89b935b6c2c6ca6c/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3cabb84349d2c21cb3bf0134573965391598301da43c85bd29d870ed7d4d8d96/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c4c829e620ef4ffd11e0016b738d848203cca680de1fa60c4f58869234aa37fc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/47a6eb317b66d069b53f2b788060022cebbb5176084f453b5ce90537d21e1190/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/92d525ef9e81898a7dbccbc31da2ac473d859fa833374b9f8d4c1913379c1661/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cd5115e2226901fab10019d7d0a3f05d1c264c97c5787e4036f2e5fe3773b924/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5f5dfa4d4d81b853f42326d2a96158368a67baf28849f127ef6eb348f2113237/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/833d7ab400140150d976605a8b5efa5d8c9e1e5fba9c4aa176ba160d63e00077/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b1ec4235e33ecf38f02608e7c46c073c7598e419c46aeef7ded462cd8bb1b89b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c96064025db0b7ef13f47e021d0cd1cc531fb8fe2b118cb29de09e8249adda50/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/d1f159c9368e589837fb71686f76998e9a24c536b4c949187daf368d4c29fe50/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/27d4949195586225db2b91985ca6e7c81647c5c623deeceb39534ea0f4238eb1/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0854384db51692415c8a094c2e7438f1af8d7e955f452966852bf93936c04c07/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/00fd5ba2409620aa899c944646a521c8e1ae31f34d23ec74ad88517870ba8d92/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/50905cc9fd44bd39b2a04078772d89a3b6da76290c17aade274d51b7bfe873ba/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/e001639e5f4f80a5becf47a292b8debf45907c0524fa62014718deb17553917a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/84633dc7967d1424a59afbed20388ab1715674710d7726062206eccc6998a54c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/e67037b4602383909ef4b0bed1d64619afa8fdca78e93500e87f28163da71b89/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/b8cb83d096f9fcdda454dd5ce6f3879fc62b96fe2faf380d6d81bf3821ff61c0/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c3b514fdacbb3ccf79f091f811980d95cb953881eb26e8302a65ea1279c1729a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/bd3b19c285073a57d48aa89af38e3e49a92c2f5bdbbafff1c7fd0d8e8a2e4bbd/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/cbed084e5c2a6489820b28b26e79691ba2562c8b548f93cc17e2ecf99eadb057/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/0737717fd0de5081022df68bb9f3c14ec31f9a5797b14e18a78ea4a97702983d/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/96cc39a2a8fda8fdc517a6a51de77ace55c7adc6632176ed76d8cd97274f78a2/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/47e4afc45503a489848c0bf692024c105df2341fbe88c84a1df2fda5860503c3/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/13d5771057b6be0b74f2c640c89afd2c73afb01cfa1e37ccb94bcce5d20a1d05/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/f0e2943d0f441bcd05e7d00402e666b193b29238ad2035e3c89e0424ad768bd2/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/5d7438f733bacefbb510514ba664fe6f16162e00f22914bb4ea37abe03489bcc/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/785da22bf43ffd472244fd5007f82adbb02518ae1b341fe1813dbf6cb671d6c4/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/1296306dfb586c5f65b88089b7493e1c5354d31dd578dd879aba852cf2ed7d2d/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/bceaecc5ea1bd4c9730fdaee8646ee593a6e1fdbaba7a6410ba5049b988828ae/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/b66e3caf1128d40092fc406a2449c8bcd288afabb4feeb3ef6b408cb3f1aeb4c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/4213a392952668c21ee3fbad8626d6678d2965a7111d527aad830de37405e27c/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a36dce2f9549512d6e0fb4f5ad3f7dc67dfb1ebdd2602902b1b81c52e7790cda/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/faeee86df990c5590a58358812f0b2dc4ea1623c7fe79d567ccf6bc1e512ab11/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/79e3eaae50d7fa5c3a5c36bae63010971e6677aee06685a48f61ea1040748dcf/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/29f932ac9e4b3feaf8aff8d6628f7790ea695f36192fb64bd8f1bec16c836461/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/0f5fbc3f661bd6f9ff119e61a82cba208ddd6303e5d9d0fe82dc4028515cdd71/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/1b9c5f7cdc1070cdefa2ef45553d2c674cec3e7a265c4d34d803baeed3996cc3/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/cefebf1119dc670d292ecd74735159641e6a44879d9cb71003feed3a2e76f817/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/4b31dbd9b1faa61035797ff904598cbe32a6680597e3e93a19be0f622fbafbc9/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/e0c4932ecea347467475b82f6412cf41215c82aa5ede76720634b53284e0581c/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/e35d7e282507dc60aeff1d8c75838bfd755ce48db8f45f60057f9e1b38a74074/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/1892b12879bc73b167e6884d94774335ac2bf0eb6b616378976bea645ae81b0c/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/e3fec0f5548a27063def4d202b94e2304c770e19c991c02d2dcc5a6695e52f70/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/2adc2f1efb7f135a7626fd2949880763c939ffec459a2c241a54ec99cf4ecd40/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/6f4b32bb0945892ae15653c30a0fe14a8559869557fa5ad63870730cd3e2d4af/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/3cd1312106efe32c25b3cba11bb824837d13c796f194cea0aec9f3ba6f3409d7/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/dec0d17d28d7b6da057db6c7ba2eed9c39b2a87ae463dea210f1d81ccf8b72d1/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/91f3aa4320e205a87a5827a4edc7fb4bc2dde951a0f3a2922d315d1bc2e6cd43/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/563c632ad01fc9554e1e6a9070e05465472e28915e068247cc6fa4c7c4481809/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/5180401060a22b68435b0de008e8e375d2ffde01f9d2ad6b74d543ac6782879d/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/b3ea8d2c34e0ac4bba59655eb3b3c5595dd082cf3d745b9242e4fba5fdfa145c/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/691ce990f3dd4566ce3b15e20179769be19c5e3e78cdc04db41c2a20490a23af/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/81ed64f5d9dbd8e53633ce99692ce4139b27ebe5b353ef3b8c8d585df03c656e/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/5ae41cb4f64757737d1e27cb8f17c934d72c0b9bb8adab2ec262817fc33f817c/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/41aace558eb93bf69e3d05b0beac18277fde238a9b3a1575e60f2a468ae13a96/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/01750cd5b825fa895627c11b4b2733ab92928110022526921907801bc8b677ac/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/05724b2e181410bedc6fd676c26853f437caaee0911b3b78d09eb9b0f6e31cd6/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/05a4aac4225236bf38342a6027c0100bebcfb6efae9c9c72c14aa616f751fdaf/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/80c0c32cb05a44012e0f4da8213f472ac6eb4c80567224970fdc2528b1577061/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/45612cc6b12644e419254fc61022c4267add43c7f1a98187f8159abf66ec5583/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ac8454b30f0625fffb0960effd4ac6a1571230d373063688fb1e097f8fb3e552/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d1887377483c6cb9f0ebbbcb00deca3793c39fff5295916499bf547c83e52cd6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/04165f3deec441a9092adb0450da8e39eb666918ccbd39c1accfbe80032dd4bd/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/754f1c13ee9ac03ce91a34aca182c9df08f3c1f71d923450513b0650e6772840/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/127ecc60176e31268220f8ff9f6037ae556230004c0ceb088d61a4abb5228ea3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e1a333631c1449d771b87faf9c4f0a8a965097bf3a2ae401cc95da0d08dde60d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/e43ab50745c38ba14afd0c640b3b12b4b4bbcd1dce5f6da76ae543268bc91e57/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/0a20604fd98f80015f99dab1383c9dc2af0bfa63809770173a8d543ecd414305/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/166fc28ae12415e2a7d8bf6c2c080910bd6151ca8d0cccf6c4c3ceccafad31f2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/e50c64631689bddddb6e64ccdd253a7744f3fc73bde2f848d32604e492ff9cd1/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 01 21:44:56.806219 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.805812    5845 manager.go:217] Machine: {Timestamp:2026-11-01 21:44:56.805344203 +0000 UTC m=+0.061523958 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072034 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec261226253deb6643cf0ae29fc9a39e SystemUUID:ec261226-253d-eb66-43cf-0ae29fc9a39e BootID:57f795f9-fd02-4d78-a3e5-b4f48f806caa Filesystems:[{Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/kube-api-access-z7ctf DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d602079-f8c5-49f7-afe1-8d6e074b8c27/volumes/kubernetes.io~projected/kube-api-access-hhc56 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95abda364a269e6982b09e0a08bde8cb4863ae2d68d0b61435ad164fd59d2cb9/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f8d10966efbd4dfdff4937a61190b749a53bc388f92aeba4688b401a36c5850/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baf3bf5f-cbda-43b8-b263-dad640f9f5a3/volumes/kubernetes.io~projected/kube-api-access-cdfrk DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:425 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:426 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~projected/kube-api-access-9g6rl DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~projected/kube-api-access-rjtfv DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:421 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dac098ba-33a8-4a99-812f-ae5561684d7c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/98dd454d-3bb0-48d9-b6ed-d907da241372/volumes/kubernetes.io~projected/kube-api-access-f9n7x DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34c29aacc4e73ff4427f636013f8270dfd399023a61566c7c2a9c9805ff19577/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~projected/kube-api-access-z9jg2 DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~projected/kube-api-access-tv58b DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7d58a7a-ace5-47e8-9d39-6ffcf04804af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~projected/kube-api-access-z9c9q DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:424 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:430 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:422 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~projected/kube-api-access-2b67s DeviceMajor:0 DeviceMinor:431 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550/userdata/shm DeviceMajor:0 DeviceMinor:432 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08cc94b0673f5a0c3d1f0fae0d8d1d955e7a9e7cccacb0a9a49b6eae69328dde/userdata/shm DeviceMajor:0 DeviceMinor:476 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6adad53666155e8835c550c28091b7d22cd6e03bf9775f67d1665dfc9a5a4a0c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:423 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9560b87304302a3503bff256b315e21a64f1730db1585e6f3609c0981a6d7973/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:427 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4538c67be1fe96ff00670fe3a1a918628d15b18c81b5b58828adbcb7dbcfe0b7/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~projected/kube-api-access-78gfv DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~projected/kube-api-access-6xdkj DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:430 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~projected/kube-api-access-f9xm7 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:362 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/890d71fb0d0173b9afbec990bac2f5d381d5bd62aaab5f2cf9b5f7d89a46d9f6/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b/volumes/kubernetes.io~projected/kube-api-access-wmp4t DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03eefce5-3634-456f-889c-3ea91faaff0d/volumes/kubernetes.io~projected/kube-api-access-4zqlt DeviceMajor:0 DeviceMinor:475 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88ed23d4d48a75c33bb922bdd643a74addf9763a35506a7a2e4dfe219c4b5e1b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cefe7ab7-6f8a-4895-af7f-747da1036866/volumes/kubernetes.io~projected/kube-api-access-7twjx DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55adb8b3-868b-4fe9-a2ee-a2de94739289/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3637cc1487d42f67c168a42a6958bb4df6918df962aae000a218b338ffea6c56/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0e6231a7b48313b055aae7a703e315f848bdaa50d16d040714a929c90e57a44/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:430 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:430 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d375a1b61f37cdd519c59184fef649460513ffb5db727cac9d50a5db6fad2059/userdata/shm DeviceMajor:0 DeviceMinor:482 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40b8c97d-537b-46e4-aa07-339c0d0b225b/volumes/kubernetes.io~projected/kube-api-access-t57cl DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f28a3119cf14a01cd345a2d69060edb4710d1f229434f3805ddba6bd94030c7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:429 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/089bb64cf6627d2c074a00d94757c0fd75bf84377d76e5d3ce6a3c69bfef2ae1/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43d867fa-5c9f-4cab-ae4e-08f4e3296a81/volumes/kubernetes.io~projected/kube-api-access-976g5 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0aea7f22b9a55681b91ec9d8449a6f2f3d01adf060cce2220eba0273f27e63b8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9761d361f4d3d013657de04a05c64f804b231ad642de67f4352e3a01cede8ca9/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f109cd89-d922-4952-b092-0360bbbbaf33/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/2ed0239d-76c2-4d3b-b168-560c022e4e3a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef221e0c-af12-43b6-9102-4a68591df51f/volumes/kubernetes.io~projected/kube-api-access-mr5kx DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90fe7ad041156002ea7d391e8e8a5651788c654fb3d0b6fad72db792f62eee56/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:474 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/defa73b4d5503aca6fa87e99bce80bfaf98cc0a160d3d27e5651d54d1d82da9a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/13c6f668-3618-4d04-9942-eab2dd9e6224/volumes/kubernetes.io~projected/kube-api-access-n2rws DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f0aef984-8fc9-46e0-8243-27b9935991a3/volumes/kubernetes.io~projected/kube-api-access-746xf DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5364a66c-4dcd-4819-a8a0-eb48a1b57f33/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fac9874b8e7c1e41020fb3fa08da7853c33692a5c70dcca1db8f7d0d62823228/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ddd68d-9250-4231-8b29-ee881e0faeae/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d688cebe03ccbac4f311c7cf2fd837d7378911e16484d768a9b28100b142643d/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cfeec369-c249-494b-b2a2-8c0753b3d17e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:428 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ba32e6c-796e-49bd-8042-db4de0b9e86b/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:08cc94b0673f5a0 MacAddress:0e:4c:19:67:72:0e Speed:10000 Mtu:8901} {Name:1a07b8108dee1c6 MacAddress:b6:4a:0b:d9:34:b6 Speed:10000 Mtu:8901} {Name:2db80b6fe51116f MacAddress:92:79:9a:77:1e:4a Speed:10000 Mtu:8901} {Name:34c29aacc4e73ff MacAddress:fa:13:1d:df:29:3f Speed:10000 Mtu:8901} {Name:4538c67be1fe96f MacAddress:3e:97:96:c1:f5:ad Speed:10000 Mtu:8901} {Name:6adad53666155e8 MacAddress:3e:b3:30:03:ae:69 Speed:10000 Mtu:8901} {Name:890d71fb0d0173b MacAddress:a2:21:38:b2:67:bb Speed:10000 Mtu:8901} {Name:8f8d10966efbd4d MacAddress:22:e2:1c:d7:38:8b Speed:10000 Mtu:8901} {Name:9560b87304302a3 MacAddress:ca:4f:b4:bd:c9:7d Speed:10000 Mtu:8901} {Name:9761d361f4d3d01 MacAddress:d6:a4:33:0f:99:82 Speed:10000 Mtu:8901} {Name:a0e6231a7b48313 MacAddress:3a:fc:7b:32:13:a0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c6:88:ad:8f Speed:0 Mtu:9001} {Name:br-int MacAddress:46:f7:12:98:5b:d3 Speed:0 Mtu:8901} {Name:d375a1b61f37cdd MacAddress:ba:32:02:f6:91:01 Speed:10000 Mtu:8901} {Name:d688cebe03ccbac MacAddress:3a:fb:73:d6:f0:cd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c6:88:ad:8f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:9c:1b:1c:e1:dd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:f4:4f:67:90:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 21:44:56.806521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806223    5845 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 21:44:56.806521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806266    5845 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 21:44:56.806588 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806539    5845 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 21:44:56.806695 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806555    5845 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-146.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 21:44:56.806736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806697    5845 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 21:44:56.806736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806705    5845 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 21:44:56.806736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806715    5845 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 21:44:56.806736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806734    5845 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 21:44:56.806842 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806803    5845 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 21:44:56.806953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.806943    5845 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 21:44:56.807063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807039    5845 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 21:44:56.807178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807082    5845 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 21:44:56.807178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807098    5845 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 21:44:56.807178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807110    5845 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 21:44:56.807178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807122    5845 kubelet.go:405] "Adding apiserver pod source"
Nov 01 21:44:56.807178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807132    5845 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 21:44:56.807988 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807970    5845 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 21:44:56.807988 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.807985    5845 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 21:44:56.808469 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.808443    5845 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 21:44:56.811132 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811117    5845 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 21:44:56.811132 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811132    5845 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 21:44:56.811230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811225    5845 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 21:44:56.811430 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811418    5845 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 21:44:56.811462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811447    5845 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 21:44:56.811539 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811530    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 21:44:56.811583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811545    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 21:44:56.811583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811554    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 21:44:56.811583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811569    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 21:44:56.811583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811574    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 21:44:56.811583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811580    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811586    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811592    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811599    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811605    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811626    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811639    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811665    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 21:44:56.811714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.811674    5845 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 21:44:56.815097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.815084    5845 server.go:1263] "Started kubelet"
Nov 01 21:44:56.815185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.815155    5845 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 21:44:56.815699 ip-10-0-135-146 systemd[1]: Started Kubernetes Kubelet.
Nov 01 21:44:56.816656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.815719    5845 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 21:44:56.816787 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.816687    5845 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 21:44:56.817018 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.816998    5845 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 21:44:56.818750 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.818732    5845 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 21:44:56.822454 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822439    5845 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 21:44:56.822579 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822511    5845 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 21:44:56.822917 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822894    5845 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 21:38:47 +0000 UTC" deadline="2028-04-05 17:28:28.575038909 +0000 UTC"
Nov 01 21:44:56.822991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822920    5845 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12499h43m31.752124561s"
Nov 01 21:44:56.822991 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:56.822951    5845 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-146.us-east-2.compute.internal\" not found"
Nov 01 21:44:56.822991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822981    5845 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 21:44:56.822991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.822988    5845 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 21:44:56.823197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.823055    5845 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 21:44:56.823972 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.823943    5845 factory.go:55] Registering systemd factory
Nov 01 21:44:56.823972 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.823970    5845 factory.go:223] Registration of the systemd container factory successfully
Nov 01 21:44:56.824600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.824572    5845 factory.go:153] Registering CRI-O factory
Nov 01 21:44:56.824600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.824592    5845 factory.go:223] Registration of the crio container factory successfully
Nov 01 21:44:56.824733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.824656    5845 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 21:44:56.824733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.824675    5845 factory.go:103] Registering Raw factory
Nov 01 21:44:56.824733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.824689    5845 manager.go:1199] Started watching for new ooms in manager
Nov 01 21:44:56.825051 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825036    5845 manager.go:319] Starting recovery of all containers
Nov 01 21:44:56.825570 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825525    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf3bf5f-cbda-43b8-b263-dad640f9f5a3" volumeName="kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk" seLinuxMountContext=""
Nov 01 21:44:56.825570 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825567    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825581    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825591    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825601    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825612    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03eefce5-3634-456f-889c-3ea91faaff0d" volumeName="kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825624    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba32e6c-796e-49bd-8042-db4de0b9e86b" volumeName="kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825637    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d602079-f8c5-49f7-afe1-8d6e074b8c27" volumeName="kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825669    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825682    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825693    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 01 21:44:56.825706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825705    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825716    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825737    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed0239d-76c2-4d3b-b168-560c022e4e3a" volumeName="kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825752    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825763    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825792    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825803    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825814    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825827    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825839    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed0239d-76c2-4d3b-b168-560c022e4e3a" volumeName="kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825850    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ddd68d-9250-4231-8b29-ee881e0faeae" volumeName="kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825861    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf3bf5f-cbda-43b8-b263-dad640f9f5a3" volumeName="kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825873    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef221e0c-af12-43b6-9102-4a68591df51f" volumeName="kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825914    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825929    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825943    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825956    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" volumeName="kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825972    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825983    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.825995    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826008    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826021    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826034    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826046    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b" volumeName="kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826057    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d58a7a-ace5-47e8-9d39-6ffcf04804af" volumeName="kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826070    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0aef984-8fc9-46e0-8243-27b9935991a3" volumeName="kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826082    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826109    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefe7ab7-6f8a-4895-af7f-747da1036866" volumeName="kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826120    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826132    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826144    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826156    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adb8b3-868b-4fe9-a2ee-a2de94739289" volumeName="kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826169    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config" seLinuxMountContext=""
Nov 01 21:44:56.826163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826180    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d58a7a-ace5-47e8-9d39-6ffcf04804af" volumeName="kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826191    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d58a7a-ace5-47e8-9d39-6ffcf04804af" volumeName="kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826203    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0aef984-8fc9-46e0-8243-27b9935991a3" volumeName="kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826215    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed0239d-76c2-4d3b-b168-560c022e4e3a" volumeName="kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826227    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" volumeName="kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826240    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826252    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826264    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826275    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826287    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826321    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b" volumeName="kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826332    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cefe7ab7-6f8a-4895-af7f-747da1036866" volumeName="kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826353    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826363    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826375    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed0239d-76c2-4d3b-b168-560c022e4e3a" volumeName="kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826386    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d602079-f8c5-49f7-afe1-8d6e074b8c27" volumeName="kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826396    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dac098ba-33a8-4a99-812f-ae5561684d7c" volumeName="kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826408    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826419    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826430    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef221e0c-af12-43b6-9102-4a68591df51f" volumeName="kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826443    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef221e0c-af12-43b6-9102-4a68591df51f" volumeName="kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826455    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826465    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826477    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826488    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826500    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826512    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826523    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826537    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826551    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826563    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" volumeName="kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826575    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" volumeName="kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826588    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826600    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7d58a7a-ace5-47e8-9d39-6ffcf04804af" volumeName="kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826612    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826630    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba32e6c-796e-49bd-8042-db4de0b9e86b" volumeName="kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826644    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40b8c97d-537b-46e4-aa07-339c0d0b225b" volumeName="kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826656    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40b8c97d-537b-46e4-aa07-339c0d0b225b" volumeName="kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826668    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826679    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61" volumeName="kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826693    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826705    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826718    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba32e6c-796e-49bd-8042-db4de0b9e86b" volumeName="kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826732    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40b8c97d-537b-46e4-aa07-339c0d0b225b" volumeName="kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826744    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98dd454d-3bb0-48d9-b6ed-d907da241372" volumeName="kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826756    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826769    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dac098ba-33a8-4a99-812f-ae5561684d7c" volumeName="kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826781    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826793    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f051ade6-cc37-418d-805c-673aaa131ae4" volumeName="kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826805    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f109cd89-d922-4952-b092-0360bbbbaf33" volumeName="kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826817    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed0239d-76c2-4d3b-b168-560c022e4e3a" volumeName="kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826828    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ba32e6c-796e-49bd-8042-db4de0b9e86b" volumeName="kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826839    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364a66c-4dcd-4819-a8a0-eb48a1b57f33" volumeName="kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826852    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adb8b3-868b-4fe9-a2ee-a2de94739289" volumeName="kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826864    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adb8b3-868b-4fe9-a2ee-a2de94739289" volumeName="kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826876    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfeec369-c249-494b-b2a2-8c0753b3d17e" volumeName="kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826888    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826899    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcb06ecb-8200-42eb-979e-d407d18b8f10" volumeName="kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826913    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c6f668-3618-4d04-9942-eab2dd9e6224" volumeName="kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826926    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55adb8b3-868b-4fe9-a2ee-a2de94739289" volumeName="kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826938    5845 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ddd68d-9250-4231-8b29-ee881e0faeae" volumeName="kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert" seLinuxMountContext=""
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826949    5845 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.826957    5845 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 21:44:56.827895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.827166    5845 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:44:56.834552 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.831459    5845 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:44:56.834552 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.832852    5845 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:44:56.834552 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.833261    5845 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 21:44:56.851530 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.851507    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:56.851767 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.851731    5845 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 21:44:56.854071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.853091    5845 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 21:44:56.854071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.853130    5845 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 21:44:56.854071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.853151    5845 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 21:44:56.854071 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:56.853189    5845 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 21:44:56.854932 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.854498    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:56.859181 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.857275    5845 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 21:44:56.870873 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870850    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="3dee586a2835bd1671154f6f0ff5b788f6a446eb3cc6c4eed2e35cf8bb01bc7c" exitCode=0
Nov 01 21:44:56.870873 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870871    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="0afc435f966c9e5f50b1cccea8a74c99ce85b90273c6cef15b1e904c53e99b0b" exitCode=0
Nov 01 21:44:56.870981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870882    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="65c2099eb421bb5f417d55637a21a08d412b1c2dbf93356f9e5dc7b32965101b" exitCode=0
Nov 01 21:44:56.870981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870890    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="6cf87a8956cb1565ec7858fbd967961b3f9c40ddd15fd10bc9db5b1287d308ed" exitCode=0
Nov 01 21:44:56.870981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870896    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="1cb0b6e0ec28eeb7c46cbc1e33a9a526d11b32fa776036d36117575441684a3d" exitCode=0
Nov 01 21:44:56.870981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.870901    5845 generic.go:356] "Generic (PLEG): container finished" podID="43d867fa-5c9f-4cab-ae4e-08f4e3296a81" containerID="eb98c7e91d0609f94f256ae6da38a440d7d2c409b4d79d77a5e1cc818635c216" exitCode=0
Nov 01 21:44:56.879347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.879328    5845 generic.go:356] "Generic (PLEG): container finished" podID="13c6f668-3618-4d04-9942-eab2dd9e6224" containerID="0f45da16f226dc92247939ef7e05945de84142b100d8bbe5b2710b0d7b55e02e" exitCode=0
Nov 01 21:44:56.897683 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.897659    5845 generic.go:356] "Generic (PLEG): container finished" podID="181995cb0fabc637e7891e096db17634" containerID="41b22f885ac1fbddb720e1994bfe4662d833d63837ed65077708d9d20ba66329" exitCode=0
Nov 01 21:44:56.901286 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.901263    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1" exitCode=0
Nov 01 21:44:56.904722 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.904701    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="4894384708fc5d0fed581faf982955a8332ee38f62787ff73db2b4862eab0025" exitCode=0
Nov 01 21:44:56.908568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.908550    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:44:56.908902 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.908883    5845 generic.go:356] "Generic (PLEG): container finished" podID="2ed0239d-76c2-4d3b-b168-560c022e4e3a" containerID="1a4bc2f8b6dca409e57fd2921b947db7a6769abfa06e54b8735fd969760d8e0f" exitCode=1
Nov 01 21:44:56.940056 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.940001    5845 manager.go:324] Recovery completed
Nov 01 21:44:56.954241 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:56.954216    5845 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 01 21:44:56.964358 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964340    5845 cpu_manager.go:226] "Starting" policy="none"
Nov 01 21:44:56.964358 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964356    5845 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 21:44:56.964513 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964375    5845 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 21:44:56.964513 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964502    5845 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964512    5845 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964529    5845 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964536    5845 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964544    5845 policy_none.go:50] "Start"
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964555    5845 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 21:44:56.964599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964564    5845 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 21:44:56.964769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964692    5845 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 21:44:56.964769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964702    5845 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 21:44:56.964769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.964708    5845 policy_none.go:44] "Start"
Nov 01 21:44:56.969485 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.969458    5845 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 21:44:56.972488 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.972473    5845 manager.go:341] "Starting Device Plugin manager"
Nov 01 21:44:56.972951 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:56.972502    5845 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 21:44:56.973024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.972955    5845 server.go:87] "Starting device plugin registration server"
Nov 01 21:44:56.973241 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973227    5845 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 21:44:56.973326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973241    5845 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 21:44:56.973378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973346    5845 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 21:44:56.973464 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973432    5845 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 21:44:56.973513 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973480    5845 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 21:44:56.973513 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973488    5845 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 21:44:56.973628 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.973544    5845 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T21:44:56.973462879Z","UUID":"b06f6b29-53de-4e4e-83d8-4e824aaacd67","Handler":null,"Name":"","Endpoint":""}
Nov 01 21:44:56.975244 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:56.975219    5845 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 21:44:56.975416 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.975269    5845 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 21:44:56.975416 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:56.975312    5845 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 21:44:57.074323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.074282    5845 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 21:44:57.076268 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.076248    5845 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 21:44:57.076391 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.076284    5845 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 21:44:57.076391 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.076318    5845 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-146.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 21:44:57.076391 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.076382    5845 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.088902 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.088880    5845 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.088975 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.088939    5845 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.155102 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155043    5845 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"]
Nov 01 21:44:57.155725 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155711    5845 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.155792 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155749    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerStarted","Data":"59eb7bcd2e9f23dac493b9d099e328109074a84d3ee33c1af29b59aeadd783bc"}
Nov 01 21:44:57.155824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155804    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerDied","Data":"41b22f885ac1fbddb720e1994bfe4662d833d63837ed65077708d9d20ba66329"}
Nov 01 21:44:57.155854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155823    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"181995cb0fabc637e7891e096db17634","Type":"ContainerStarted","Data":"c57a817c4205c0f6d67ce7562647a5066acf606ec7d9b6fec108081af84ca364"}
Nov 01 21:44:57.155909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155895    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"ec31c2ba0ee2bf174eddd7fce91e152b","Type":"ContainerStarted","Data":"56240583552aaef3ef5aa70d5de9b7b0633f3fa081072db778f49b9e89996bd9"}
Nov 01 21:44:57.155940 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155920    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal" event={"ID":"ec31c2ba0ee2bf174eddd7fce91e152b","Type":"ContainerStarted","Data":"5595e878c7316d3e95a4c4663b631aee2c22ec1cffb77d2b6d2854abce890348"}
Nov 01 21:44:57.155988 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.155977    5845 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.168124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.168109    5845 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:44:57.168188 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:57.168146    5845 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.170938 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.170924    5845 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:44:57.170993 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:57.170959    5845 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.235258 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.235234    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.235360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.235261    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.235360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.235278    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335803 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335776    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335898 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335814    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335898 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335837    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335898 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335884    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec31c2ba0ee2bf174eddd7fce91e152b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"ec31c2ba0ee2bf174eddd7fce91e152b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335983 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335894    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.335983 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.335884    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181995cb0fabc637e7891e096db17634-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" (UID: \"181995cb0fabc637e7891e096db17634\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.807834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.807808    5845 apiserver.go:52] "Watching apiserver"
Nov 01 21:44:57.825689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.825665    5845 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 21:44:57.826381 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.826357    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9wtgk","openshift-network-operator/iptables-alerter-bdpxm","openshift-insights/insights-runtime-extractor-bgzv8","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-f5s44","openshift-image-registry/node-ca-r7hr8","kube-system/global-pull-secret-syncer-f4s8z","openshift-dns/node-resolver-r86vn","openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2","openshift-cluster-node-tuning-operator/tuned-6wp5h","openshift-multus/multus-additional-cni-plugins-xs9kv","openshift-multus/network-metrics-daemon-bcxfp","openshift-dns/dns-default-9jp45","openshift-ingress-canary/ingress-canary-sn9hx","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-hmnzz","openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9","openshift-multus/multus-8gkcc","openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn","openshift-monitoring/metrics-server-6f99686fb5-2wbs5","openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x","openshift-ovn-kubernetes/ovnkube-node-zw7wh","kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"]
Nov 01 21:44:57.831613 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.831590    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.831748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.831615    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4w95n\""
Nov 01 21:44:57.831748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.831650    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.831748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.831704    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 21:44:57.832844 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.832812    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.832945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.832900    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.832945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.832914    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 21:44:57.832945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.832927    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hnwx2\""
Nov 01 21:44:57.833106 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.833056    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 21:44:57.835186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835162    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 21:44:57.835186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835166    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 21:44:57.835334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835166    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.835334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835233    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 21:44:57.835334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835243    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.835555 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835540    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.835611 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835597    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.835658 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835642    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2cpnk\""
Nov 01 21:44:57.835695 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835648    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ww679\""
Nov 01 21:44:57.835725 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.835701    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.836814 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.836794    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s2qsr\""
Nov 01 21:44:57.836814 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.836796    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.836927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.836901    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 21:44:57.837038 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.837026    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 21:44:57.837068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.837055    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6vk2z\""
Nov 01 21:44:57.837131 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.837121    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 21:44:57.837161 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.837103    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ntb2c\""
Nov 01 21:44:57.839417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.839397    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.918063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.918037    5845 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.918201 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.918165    5845 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.925799 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.925780    5845 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 21:44:57.929882 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.929865    5845 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:44:57.929943 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:57.929902    5845 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.930476 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.930461    5845 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 21:44:57.930551 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:44:57.930499    5845 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-146.us-east-2.compute.internal"
Nov 01 21:44:57.933095 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933074    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 21:44:57.933179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933075    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.933179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933079    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.933179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933146    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.933332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933187    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.933434 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933416    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.933528 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933465    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 21:44:57.933628 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933612    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.933670 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.933626    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.934431 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934415    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 21:44:57.934497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934415    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7lpbt\""
Nov 01 21:44:57.934743 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934730    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 21:44:57.934793 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934740    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 21:44:57.934843 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934790    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 21:44:57.934881 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934857    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 21:44:57.934956 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934940    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 21:44:57.935035 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.934950    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 21:44:57.935091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935055    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 21:44:57.935091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935063    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 21:44:57.935163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935087    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 21:44:57.935780 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935766    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 21:44:57.935874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935859    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 21:44:57.935924 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935891    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 21:44:57.935924 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935894    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 21:44:57.935924 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935918    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 21:44:57.936029 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.935930    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 21:44:57.936380 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936366    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 21:44:57.936495 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936481    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 21:44:57.936539 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936522    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 21:44:57.936586 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936522    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-59b8p\""
Nov 01 21:44:57.936586 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936550    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 21:44:57.936586 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936572    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 21:44:57.936678 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.936587    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nrwmx\""
Nov 01 21:44:57.937231 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937218    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.937271 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937235    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 21:44:57.937662 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937647    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 21:44:57.937734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937667    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.937734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937701    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937731    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937789    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937797    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937802    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937793    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937824    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 21:44:57.937834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937833    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xhnrl\""
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937839    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937889    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937896    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bzsh4\""
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937931    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937962    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.937968    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938012    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938061 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938043    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938076    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938138    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938159    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938183    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938231    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938257    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938265    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdkj\" (UniqueName: \"kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938403    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938409    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938453    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938490    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938531    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938567    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938602    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938566    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d602079-f8c5-49f7-afe1-8d6e074b8c27-serviceca\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938632    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938682    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938716    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bpkb5\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938705    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938738    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938735    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938748    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938684    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938771    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938764    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ncxjw\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938785    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-glcdl\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938793    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938807    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938822    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938836    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:57.938878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938865    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976g5\" (UniqueName: \"kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938909    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.938935    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939009    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939026    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939093    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939151    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-daemon-config\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939175    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6rl\" (UniqueName: \"kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939213    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939239    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-config\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939241    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939279    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939350    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939353    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939388    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939406    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746xf\" (UniqueName: \"kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939439    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939462    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939471    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5crl4\""
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939512    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939526    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40b8c97d-537b-46e4-aa07-339c0d0b225b-cni-binary-copy\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939532    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939543    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939571    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939600    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.939638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939624    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939662    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939699    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939721    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939749    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939775    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939777    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cfeec369-c249-494b-b2a2-8c0753b3d17e-audit-log\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939830    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939848    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939866    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939869    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939921    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939933    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-accelerators-collector-config\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.939949    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940009    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940031    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xhnwq\""
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940064    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940082    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940067    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940126    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940151    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940179    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940185    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940196    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-original-pull-secret\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940222    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940257    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940281    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940287    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940345    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940389    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940408    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940418    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmp4t\" (UniqueName: \"kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940430    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940446    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940473    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940584    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.940596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940608    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5kx\" (UniqueName: \"kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940624    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940662    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940675    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940682    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-chkoj1b7amdmk\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940690    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940713    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-tmp\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940737    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940750    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940782    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940844    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940888    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940902    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pth5p\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940893    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-56hmk\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940933    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940939    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940844    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6pn6l\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940957    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940975    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940947    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941026    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vh4k\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.940991    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941060    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941083    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941116    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941139    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941139    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-binary-copy\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941158    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjtfv\" (UniqueName: \"kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941175    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941199    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941214    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941240    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941272    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941278    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941320    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941347    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xm7\" (UniqueName: \"kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941373    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941488    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941518    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941555    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941558    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941587    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941606    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941642    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941664    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhrgg\""
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941724    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twjx\" (UniqueName: \"kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941759    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941791    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941813    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941831    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941850    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-textfile\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.941899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941892    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f109cd89-d922-4952-b092-0360bbbbaf33-tmp\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941946    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941966    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.941986    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942020    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942045    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942052    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/55adb8b3-868b-4fe9-a2ee-a2de94739289-metrics-tls\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942063    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942081    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942114    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942142    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942171    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942177    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942206    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942211    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942234    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-client-certs\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942230    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942334    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-68sovd0l9bojt\""
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942347    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jg2\" (UniqueName: \"kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942409    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942576    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942595    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nk9sj\""
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942581    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942617    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-26v4dnni8p4ij\""
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942632    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942657    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942673    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942689    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv58b\" (UniqueName: \"kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942704    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942746    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942752    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942786    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942803    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942815    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942824    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c9q\" (UniqueName: \"kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942841    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942855    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942867    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovnkube-script-lib\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942877    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942889    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-data-volume\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942893    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942916    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942931    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942947    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942964    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.942980    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943037    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57cl\" (UniqueName: \"kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943054    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943119    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943125    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-metrics-certs\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943137    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943155    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rws\" (UniqueName: \"kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943156    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-tmp-dir\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943171    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943190    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943207    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943228    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943245    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943262    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943281    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc56\" (UniqueName: \"kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943283    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943311    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943346    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/55adb8b3-868b-4fe9-a2ee-a2de94739289-config-volume\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943398    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943427    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943455    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943488    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfrk\" (UniqueName: \"kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943521    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943537    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-client-ca-bundle\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943538    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2ddd68d-9250-4231-8b29-ee881e0faeae-cert\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943624    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943653    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943704    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943731    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gfv\" (UniqueName: \"kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943747    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943759    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943790    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943805    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943916    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55adb8b3-868b-4fe9-a2ee-a2de94739289-tmp-dir\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943916    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943960    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:57.944064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.943999    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944410    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cefe7ab7-6f8a-4895-af7f-747da1036866-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944473    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944531    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944547    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944565    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2ed0239d-76c2-4d3b-b168-560c022e4e3a-env-overrides\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944608    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944782    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.944871    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945011    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945052    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2ed0239d-76c2-4d3b-b168-560c022e4e3a-ovn-node-metrics-cert\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945067    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945115    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ef221e0c-af12-43b6-9102-4a68591df51f-etc-tuned\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945177    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945326    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945488    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945542    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945681    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945725    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945766    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945374    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ba32e6c-796e-49bd-8042-db4de0b9e86b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.945928    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9n7x\" (UniqueName: \"kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946327    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946379    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946422    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946419    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-tls\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946708    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946737    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cfeec369-c249-494b-b2a2-8c0753b3d17e-metrics-server-audit-profiles\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.946796    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.947000    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.947020    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cfeec369-c249-494b-b2a2-8c0753b3d17e-secret-metrics-server-tls\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.947285    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dac098ba-33a8-4a99-812f-ae5561684d7c-konnectivity-ca\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:57.947639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.947456    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dac098ba-33a8-4a99-812f-ae5561684d7c-agent-certs\") pod \"konnectivity-agent-9wtgk\" (UID: \"dac098ba-33a8-4a99-812f-ae5561684d7c\") " pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:44:57.949430 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.948407    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.949430 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.948497    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.949625 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.949603    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0aef984-8fc9-46e0-8243-27b9935991a3-iptables-alerter-script\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:57.949857 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.949820    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.950239 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.950211    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.950639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.950618    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.950744 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.950723    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.950832 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.950812    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f109cd89-d922-4952-b092-0360bbbbaf33-secret-grpc-tls\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.951477 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.951446    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f109cd89-d922-4952-b092-0360bbbbaf33-metrics-client-ca\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.951698 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.951603    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ba32e6c-796e-49bd-8042-db4de0b9e86b-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.951698 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.951592    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13c6f668-3618-4d04-9942-eab2dd9e6224-metrics-client-ca\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.952093 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952073    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 21:44:57.952093 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952074    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 21:44:57.952271 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952254    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952543    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952763    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976g5\" (UniqueName: \"kubernetes.io/projected/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-kube-api-access-976g5\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.952919    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdkj\" (UniqueName: \"kubernetes.io/projected/f109cd89-d922-4952-b092-0360bbbbaf33-kube-api-access-6xdkj\") pod \"thanos-querier-fb8c8bd9c-6mv4x\" (UID: \"f109cd89-d922-4952-b092-0360bbbbaf33\") " pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953004    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746xf\" (UniqueName: \"kubernetes.io/projected/f0aef984-8fc9-46e0-8243-27b9935991a3-kube-api-access-746xf\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953021    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjtfv\" (UniqueName: \"kubernetes.io/projected/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-kube-api-access-rjtfv\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:57.953048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953022    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xm7\" (UniqueName: \"kubernetes.io/projected/55adb8b3-868b-4fe9-a2ee-a2de94739289-kube-api-access-f9xm7\") pod \"dns-default-9jp45\" (UID: \"55adb8b3-868b-4fe9-a2ee-a2de94739289\") " pod="openshift-dns/dns-default-9jp45"
Nov 01 21:44:57.953287 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953122    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6rl\" (UniqueName: \"kubernetes.io/projected/cfeec369-c249-494b-b2a2-8c0753b3d17e-kube-api-access-9g6rl\") pod \"metrics-server-6f99686fb5-2wbs5\" (UID: \"cfeec369-c249-494b-b2a2-8c0753b3d17e\") " pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:44:57.953287 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953273    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5kx\" (UniqueName: \"kubernetes.io/projected/ef221e0c-af12-43b6-9102-4a68591df51f-kube-api-access-mr5kx\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:57.953362 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953283    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twjx\" (UniqueName: \"kubernetes.io/projected/cefe7ab7-6f8a-4895-af7f-747da1036866-kube-api-access-7twjx\") pod \"monitoring-plugin-77f4f76594-kwlgq\" (UID: \"cefe7ab7-6f8a-4895-af7f-747da1036866\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:44:57.953475 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953453    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmp4t\" (UniqueName: \"kubernetes.io/projected/bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b-kube-api-access-wmp4t\") pod \"network-metrics-daemon-bcxfp\" (UID: \"bedd4f98-22d8-4bbf-8e7a-6c54d1504e1b\") " pod="openshift-multus/network-metrics-daemon-bcxfp"
Nov 01 21:44:57.953589 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953573    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv58b\" (UniqueName: \"kubernetes.io/projected/2ed0239d-76c2-4d3b-b168-560c022e4e3a-kube-api-access-tv58b\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:57.953637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953574    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jg2\" (UniqueName: \"kubernetes.io/projected/5364a66c-4dcd-4819-a8a0-eb48a1b57f33-kube-api-access-z9jg2\") pod \"kube-state-metrics-5dc8d7d89b-97sfn\" (UID: \"5364a66c-4dcd-4819-a8a0-eb48a1b57f33\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-97sfn"
Nov 01 21:44:57.953827 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.953810    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.954156 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.954138    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.954963 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.954947    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57cl\" (UniqueName: \"kubernetes.io/projected/40b8c97d-537b-46e4-aa07-339c0d0b225b-kube-api-access-t57cl\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:57.957157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.957135    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc56\" (UniqueName: \"kubernetes.io/projected/8d602079-f8c5-49f7-afe1-8d6e074b8c27-kube-api-access-hhc56\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:57.957238 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.957162    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfrk\" (UniqueName: \"kubernetes.io/projected/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-kube-api-access-cdfrk\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:57.957325 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.957306    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c9q\" (UniqueName: \"kubernetes.io/projected/2ba32e6c-796e-49bd-8042-db4de0b9e86b-kube-api-access-z9c9q\") pod \"openshift-state-metrics-7bd7489957-8vxv9\" (UID: \"2ba32e6c-796e-49bd-8042-db4de0b9e86b\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8vxv9"
Nov 01 21:44:57.957643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.957622    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqlt\" (UniqueName: \"kubernetes.io/projected/03eefce5-3634-456f-889c-3ea91faaff0d-kube-api-access-4zqlt\") pod \"network-check-target-f5s44\" (UID: \"03eefce5-3634-456f-889c-3ea91faaff0d\") " pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:44:57.958990 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.958969    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:44:57.960171 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.960151    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rws\" (UniqueName: \"kubernetes.io/projected/13c6f668-3618-4d04-9942-eab2dd9e6224-kube-api-access-n2rws\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:57.960279 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.960257    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9n7x\" (UniqueName: \"kubernetes.io/projected/98dd454d-3bb0-48d9-b6ed-d907da241372-kube-api-access-f9n7x\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:57.960454 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.960434    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:57.960548 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.960534    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gfv\" (UniqueName: \"kubernetes.io/projected/a2ddd68d-9250-4231-8b29-ee881e0faeae-kube-api-access-78gfv\") pod \"ingress-canary-sn9hx\" (UID: \"a2ddd68d-9250-4231-8b29-ee881e0faeae\") " pod="openshift-ingress-canary/ingress-canary-sn9hx"
Nov 01 21:44:57.963322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:57.963291    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:44:58.048167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048145    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.048277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048177    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048202    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048239    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048253    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-hostroot\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048279    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-socket-dir-parent\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048244    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-var-lib-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048279    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048332    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-kubernetes\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048340    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-multus-certs\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048358    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048378    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048394    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048407    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048416    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysconfig\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048442    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048448    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-systemd-units\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048460    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048460    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-kubelet-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048477    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d602079-f8c5-49f7-afe1-8d6e074b8c27-host\") pod \"node-ca-r7hr8\" (UID: \"8d602079-f8c5-49f7-afe1-8d6e074b8c27\") " pod="openshift-image-registry/node-ca-r7hr8"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048483    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048494    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-run\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.048503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048504    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048533    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048566    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048600    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-etc-kubernetes\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048610    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048635    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048650    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048665    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048709    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048709    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-systemd\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048724    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048724    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048729    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-node-log\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048755    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-multus-conf-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048769    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-slash\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048774    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-sys-fs\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048759    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-etc-selinux\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048799    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048817    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048834    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048836    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-systemd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048859    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048847    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-sys\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048866    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-kubelet\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048881    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048910    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048925    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048949    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-netd\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048947    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048976    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048977    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-root\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048977    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-cnibin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.048992    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049006    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049016    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-host\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049019    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-k8s-cni-cncf-io\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049048    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049049    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-run-netns\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049089    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-cnibin\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049094    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-modprobe-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049130    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049145    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-d\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.049124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049152    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049170    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049177    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-cni-bin\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049186    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049202    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049210    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-kubelet-config\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049221    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049244    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d867fa-5c9f-4cab-ae4e-08f4e3296a81-system-cni-dir\") pod \"multus-additional-cni-plugins-xs9kv\" (UID: \"43d867fa-5c9f-4cab-ae4e-08f4e3296a81\") " pod="openshift-multus/multus-additional-cni-plugins-xs9kv"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049247    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049251    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-etc-sysctl-conf\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049264    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-device-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049271    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049258    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-ovn\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049267    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-bin\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049323    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049344    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61-dbus\") pod \"global-pull-secret-syncer-f4s8z\" (UID: \"c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61\") " pod="kube-system/global-pull-secret-syncer-f4s8z"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049350    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049348    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-run-netns\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049407    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049423    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-lib-modules\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049428    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049444    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049447    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-kubelet\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049459    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049479    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049475    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-sys\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049505    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049528    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-socket-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049541    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13c6f668-3618-4d04-9942-eab2dd9e6224-node-exporter-wtmp\") pod \"node-exporter-hmnzz\" (UID: \"13c6f668-3618-4d04-9942-eab2dd9e6224\") " pod="openshift-monitoring/node-exporter-hmnzz"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049530    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-etc-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049545    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049581    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-run-openvswitch\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049592    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049609    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049611    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049628    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049660    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7d58a7a-ace5-47e8-9d39-6ffcf04804af-crio-socket\") pod \"insights-runtime-extractor-bgzv8\" (UID: \"e7d58a7a-ace5-47e8-9d39-6ffcf04804af\") " pod="openshift-insights/insights-runtime-extractor-bgzv8"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049664    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049668    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98dd454d-3bb0-48d9-b6ed-d907da241372-registration-dir\") pod \"aws-ebs-csi-driver-node-98dc2\" (UID: \"98dd454d-3bb0-48d9-b6ed-d907da241372\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049685    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2ed0239d-76c2-4d3b-b168-560c022e4e3a-log-socket\") pod \"ovnkube-node-zw7wh\" (UID: \"2ed0239d-76c2-4d3b-b168-560c022e4e3a\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049688    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049710    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aef984-8fc9-46e0-8243-27b9935991a3-host-slash\") pod \"iptables-alerter-bdpxm\" (UID: \"f0aef984-8fc9-46e0-8243-27b9935991a3\") " pod="openshift-network-operator/iptables-alerter-bdpxm"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049721    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-host-var-lib-cni-multus\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049724    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049744    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049760    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049807    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/baf3bf5f-cbda-43b8-b263-dad640f9f5a3-hosts-file\") pod \"node-resolver-r86vn\" (UID: \"baf3bf5f-cbda-43b8-b263-dad640f9f5a3\") " pod="openshift-dns/node-resolver-r86vn"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049813    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40b8c97d-537b-46e4-aa07-339c0d0b225b-system-cni-dir\") pod \"multus-8gkcc\" (UID: \"40b8c97d-537b-46e4-aa07-339c0d0b225b\") " pod="openshift-multus/multus-8gkcc"
Nov 01 21:44:58.050332 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:58.049824    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef221e0c-af12-43b6-9102-4a68591df51f-var-lib-kubelet\") pod \"tuned-6wp5h\" (UID: \"ef221e0c-af12-43b6-9102-4a68591df51f\") " pod="openshift-cluster-node-tuning-operator/tuned-6wp5h"
Nov 01 21:44:59.801270 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:44:59.801241    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:45:00.391773 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:00.391740    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jp45"
Nov 01 21:45:00.445511 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:00.445483    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 21:45:00.480854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:00.480830    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:00.645591 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:00.645520    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x"
Nov 01 21:45:02.340815 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:02.340784    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:45:02.341267 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:02.340979    5845 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 21:45:02.630642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:02.630571    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 21:45:03.928727 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:03.928690    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:45:04.814737 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:04.814705    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:45:08.054509 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:08.054432    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh"
Nov 01 21:45:12.817809 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.817773    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818192    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="alertmanager" containerID="cri-o://14bb91ca455545f819c89a223b08e68dfa01732644e524f51f42fa844cd0b30c" gracePeriod=120
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818204    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-metric" containerID="cri-o://fa85cc7366974bca56ad37f90cef4358840931f34f82404cafcb7f78a8b130c4" gracePeriod=120
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818245    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-web" containerID="cri-o://49d1775adc0b211c4f1eb6713e074144f812c4b976e9c34eb676ed602e151002" gracePeriod=120
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818245    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="prom-label-proxy" containerID="cri-o://bc6b0ac69c6d55b2ff21f397f4db1eba019f29860fa2cb822c6a7056b93b1807" gracePeriod=120
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818275    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="config-reloader" containerID="cri-o://fee4bbeae741b09de7c1641db1d21a33554b9f36b028ec6ac84a832f04ebfc9d" gracePeriod=120
Nov 01 21:45:12.818371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.818271    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy" containerID="cri-o://c1b79eec21aca3e12b9eac1ae07a74b182431bf9dd6590fc990158c3b29f3b7a" gracePeriod=120
Nov 01 21:45:12.974467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974440    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="bc6b0ac69c6d55b2ff21f397f4db1eba019f29860fa2cb822c6a7056b93b1807" exitCode=0
Nov 01 21:45:12.974467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974463    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="fa85cc7366974bca56ad37f90cef4358840931f34f82404cafcb7f78a8b130c4" exitCode=0
Nov 01 21:45:12.974467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974469    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="c1b79eec21aca3e12b9eac1ae07a74b182431bf9dd6590fc990158c3b29f3b7a" exitCode=0
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974474    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="fee4bbeae741b09de7c1641db1d21a33554b9f36b028ec6ac84a832f04ebfc9d" exitCode=0
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974480    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="14bb91ca455545f819c89a223b08e68dfa01732644e524f51f42fa844cd0b30c" exitCode=0
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974514    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"bc6b0ac69c6d55b2ff21f397f4db1eba019f29860fa2cb822c6a7056b93b1807"}
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974575    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"fa85cc7366974bca56ad37f90cef4358840931f34f82404cafcb7f78a8b130c4"}
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974602    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"c1b79eec21aca3e12b9eac1ae07a74b182431bf9dd6590fc990158c3b29f3b7a"}
Nov 01 21:45:12.974624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974626    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"fee4bbeae741b09de7c1641db1d21a33554b9f36b028ec6ac84a832f04ebfc9d"}
Nov 01 21:45:12.974799 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:12.974649    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"14bb91ca455545f819c89a223b08e68dfa01732644e524f51f42fa844cd0b30c"}
Nov 01 21:45:13.558163 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:13.558126    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9wtgk"
Nov 01 21:45:13.981548 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:13.981519    5845 generic.go:356] "Generic (PLEG): container finished" podID="f051ade6-cc37-418d-805c-673aaa131ae4" containerID="49d1775adc0b211c4f1eb6713e074144f812c4b976e9c34eb676ed602e151002" exitCode=0
Nov 01 21:45:13.981891 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:13.981575    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"49d1775adc0b211c4f1eb6713e074144f812c4b976e9c34eb676ed602e151002"}
Nov 01 21:45:14.060157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.060139    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:14.186518 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186452    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186518 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186495    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186524    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186550    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186581    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186608    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186631    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186659    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186686    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.186734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186724    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.187125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186750    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.187125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186804    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.187125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.186834    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") pod \"f051ade6-cc37-418d-805c-673aaa131ae4\" (UID: \"f051ade6-cc37-418d-805c-673aaa131ae4\") "
Nov 01 21:45:14.187125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.187044    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:14.187125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.187076    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:45:14.187408 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.187166    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-metrics-client-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.187408 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.187188    5845 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-main-db\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.187682 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.187658    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:14.189705 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.189656    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.190695 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.190658    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.190807 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.190735    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.190955 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.190912    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:45:14.191074 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.190959    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.191074 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.191003    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:45:14.191421 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.191397    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "kube-api-access-z7ctf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:45:14.191536 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.191516    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.195820 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.195797    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.202103 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.202085    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config" pod "f051ade6-cc37-418d-805c-673aaa131ae4" (UID: "f051ade6-cc37-418d-805c-673aaa131ae4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:14.288073 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288053    5845 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288073 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288073    5845 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f051ade6-cc37-418d-805c-673aaa131ae4-config-out\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288083    5845 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f051ade6-cc37-418d-805c-673aaa131ae4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288093    5845 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-tls-assets\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288103    5845 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-web-config\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288112    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ctf\" (UniqueName: \"kubernetes.io/projected/f051ade6-cc37-418d-805c-673aaa131ae4-kube-api-access-z7ctf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288124    5845 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288132    5845 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288141    5845 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-main-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288150    5845 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-cluster-tls-config\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.288191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.288159    5845 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f051ade6-cc37-418d-805c-673aaa131ae4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:14.988107 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.988080    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f051ade6-cc37-418d-805c-673aaa131ae4","Type":"ContainerDied","Data":"2db80b6fe51116f5fe2546926e47343d31f70e93fef3963f40abb43c7057f9c7"}
Nov 01 21:45:14.988479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.988141    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:14.988479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.988157    5845 scope.go:122] "RemoveContainer" containerID="bc6b0ac69c6d55b2ff21f397f4db1eba019f29860fa2cb822c6a7056b93b1807"
Nov 01 21:45:14.995793 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:14.995775    5845 scope.go:122] "RemoveContainer" containerID="fa85cc7366974bca56ad37f90cef4358840931f34f82404cafcb7f78a8b130c4"
Nov 01 21:45:15.002645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.002628    5845 scope.go:122] "RemoveContainer" containerID="c1b79eec21aca3e12b9eac1ae07a74b182431bf9dd6590fc990158c3b29f3b7a"
Nov 01 21:45:15.009063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.009044    5845 scope.go:122] "RemoveContainer" containerID="49d1775adc0b211c4f1eb6713e074144f812c4b976e9c34eb676ed602e151002"
Nov 01 21:45:15.011248 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.011193    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:15.015205 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.015184    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:15.016798 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.016782    5845 scope.go:122] "RemoveContainer" containerID="fee4bbeae741b09de7c1641db1d21a33554b9f36b028ec6ac84a832f04ebfc9d"
Nov 01 21:45:15.023095 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.023079    5845 scope.go:122] "RemoveContainer" containerID="14bb91ca455545f819c89a223b08e68dfa01732644e524f51f42fa844cd0b30c"
Nov 01 21:45:15.029728 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.029712    5845 scope.go:122] "RemoveContainer" containerID="4894384708fc5d0fed581faf982955a8332ee38f62787ff73db2b4862eab0025"
Nov 01 21:45:15.045664 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045645    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:15.045902 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045890    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-metric"
Nov 01 21:45:15.045943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045904    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-metric"
Nov 01 21:45:15.045943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045915    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="config-reloader"
Nov 01 21:45:15.045943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045920    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="config-reloader"
Nov 01 21:45:15.045943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045932    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-web"
Nov 01 21:45:15.045943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045937    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-web"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045953    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="alertmanager"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045958    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="alertmanager"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045965    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="prom-label-proxy"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045969    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="prom-label-proxy"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045979    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045984    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045991    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="init-config-reloader"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.045995    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="init-config-reloader"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046035    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="init-config-reloader"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046043    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-metric"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046049    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="config-reloader"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046055    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy-web"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046062    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="alertmanager"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046067    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="prom-label-proxy"
Nov 01 21:45:15.046082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.046073    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" containerName="kube-rbac-proxy"
Nov 01 21:45:15.053684 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.053664    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.059863 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059837    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 21:45:15.059981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059841    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 21:45:15.059981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059846    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 21:45:15.059981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059855    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 21:45:15.059981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059864    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 21:45:15.059981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.059898    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 21:45:15.060202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.060027    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 21:45:15.060202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.060035    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vhrgg\""
Nov 01 21:45:15.061815 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.061797    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 21:45:15.062116 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.062101    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 21:45:15.066439 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.066418    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:15.194134 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194062    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194134 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194096    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kbk\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-kube-api-access-s5kbk\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194134 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194119    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-config-out\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194134 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194135    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194199    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194245    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194351    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194378    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-config-volume\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194402    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194487    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194508    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194611 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194537    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-web-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.194611 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.194572    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295500 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295470    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-web-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295504    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295525    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295571    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kbk\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-kube-api-access-s5kbk\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295627    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-config-out\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295788 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295757    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295843 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295797    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295843 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295826    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295869    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295910    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-config-volume\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.295945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295907    5845 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 21:45:15.296092 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295948    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.296092 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.295991    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.296092 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.296025    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.296236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.296137    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.296792 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.296761    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.296993 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.296961    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/051fd871-015f-445a-8314-b557c25f0443-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.298880 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.298850    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-config-volume\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.298880 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.298871    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299031 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.298980    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-tls-assets\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299089 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.299049    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299375 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.299346    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299485 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.299377    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299485 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.299439    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.299618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.299546    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/051fd871-015f-445a-8314-b557c25f0443-config-out\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.300725 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.300701    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/051fd871-015f-445a-8314-b557c25f0443-web-config\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.307936 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.307913    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kbk\" (UniqueName: \"kubernetes.io/projected/051fd871-015f-445a-8314-b557c25f0443-kube-api-access-s5kbk\") pod \"alertmanager-main-0\" (UID: \"051fd871-015f-445a-8314-b557c25f0443\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.371232 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.371204    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 21:45:15.653286 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.653251    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 21:45:15.656924 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:45:15.656894    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051fd871_015f_445a_8314_b557c25f0443.slice/crio-b7ea7d505880b6c2b7d0374213ce9f0aedf24887e1d6390162947e2787cea8fd WatchSource:0}: Error finding container b7ea7d505880b6c2b7d0374213ce9f0aedf24887e1d6390162947e2787cea8fd: Status 404 returned error can't find the container with id b7ea7d505880b6c2b7d0374213ce9f0aedf24887e1d6390162947e2787cea8fd
Nov 01 21:45:15.992758 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.992729    5845 generic.go:356] "Generic (PLEG): container finished" podID="051fd871-015f-445a-8314-b557c25f0443" containerID="54d2f589d899b31199a8f0d07c95e17825a67650e077f9b37b6c6090186a68ff" exitCode=0
Nov 01 21:45:15.993114 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.992818    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerDied","Data":"54d2f589d899b31199a8f0d07c95e17825a67650e077f9b37b6c6090186a68ff"}
Nov 01 21:45:15.993114 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:15.992941    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"b7ea7d505880b6c2b7d0374213ce9f0aedf24887e1d6390162947e2787cea8fd"}
Nov 01 21:45:16.858709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.858676    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f051ade6-cc37-418d-805c-673aaa131ae4" path="/var/lib/kubelet/pods/f051ade6-cc37-418d-805c-673aaa131ae4/volumes"
Nov 01 21:45:16.999520 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999483    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"e51fb9d20f1795ab169dd8ff6a1505a92c06fdfe52a3d553ccbb59c99d8319fa"}
Nov 01 21:45:16.999874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999560    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"db5c8c4ee6a066849bb76be9d70faf31e99a4023e5a494ba5cde66fcaac1234f"}
Nov 01 21:45:16.999874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999609    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"c3abade7a58ff25adf57d83ada5214026b21163c969c654ffea2eb8c27f7aab5"}
Nov 01 21:45:16.999874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999648    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"5611d342b3ded65af822a648b9738ff2261ca63dd2588d69a87d6acf2d2ddfd5"}
Nov 01 21:45:16.999874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999681    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"2082ecefcde1c27936b8634ddb659a9dd4bf1d5e65ffa5a285998b6fb9fe7e84"}
Nov 01 21:45:16.999874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:16.999716    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"051fd871-015f-445a-8314-b557c25f0443","Type":"ContainerStarted","Data":"21aba90ac170bf0f605c4776b76d48ab8e290b9c247be44d1d20b3e23086f2a6"}
Nov 01 21:45:17.005748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.005712    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sn9hx_a2ddd68d-9250-4231-8b29-ee881e0faeae/serve-healthcheck-canary/0.log"
Nov 01 21:45:17.035058 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.035010    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.023039575 podStartE2EDuration="2.034997793s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.958218ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:45:17.032984394 +0000 UTC m=+20.289164195" watchObservedRunningTime="2026-11-01 21:45:17.034997793 +0000 UTC m=+20.291177553"
Nov 01 21:45:17.260978 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.260942    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261285    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="prometheus" containerID="cri-o://76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01" gracePeriod=600
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261326    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-web" containerID="cri-o://b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5" gracePeriod=600
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261350    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229" gracePeriod=600
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261358    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="config-reloader" containerID="cri-o://54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1" gracePeriod=600
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261326    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="thanos-sidecar" containerID="cri-o://183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0" gracePeriod=600
Nov 01 21:45:17.261471 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:17.261287    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy" containerID="cri-o://3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19" gracePeriod=600
Nov 01 21:45:18.007130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007103    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19" exitCode=0
Nov 01 21:45:18.007130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007126    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0" exitCode=0
Nov 01 21:45:18.007130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007134    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1" exitCode=0
Nov 01 21:45:18.007512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007140    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01" exitCode=0
Nov 01 21:45:18.007512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007173    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"}
Nov 01 21:45:18.007512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007237    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"}
Nov 01 21:45:18.007512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007263    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"}
Nov 01 21:45:18.007512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.007287    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"}
Nov 01 21:45:18.502928 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.502904    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:18.626315 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626220    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626315 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626264    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626531 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626322    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626531 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626487    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626641 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626581    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626647    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626724 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626686    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626765 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626719    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626819 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626767    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626819 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626794    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626891 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626825    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626891 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626866    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626896    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626927    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626962    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.626977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626958    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.627175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.626995    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.627175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627036    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.627175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627069    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.627175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627118    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.627175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627148    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") pod \"dcb06ecb-8200-42eb-979e-d407d18b8f10\" (UID: \"dcb06ecb-8200-42eb-979e-d407d18b8f10\") "
Nov 01 21:45:18.627462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627435    5845 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.627728 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627697    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.628021 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.627994    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.628103 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.628042    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.628515 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.628357    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:45:18.628515 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.628421    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.629333 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.629181    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:45:18.629333 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.629288    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.629700 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.629515    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.629700 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.629537    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.629818 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.629780    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:45:18.630576 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.630506    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.630657 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.630610    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "kube-api-access-2b67s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:45:18.630774 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.630750    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.630904 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.630884    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.631475 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.631454    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.631969 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.631949    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.632125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.632101    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.632161 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.632136    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:45:18.641092 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.641073    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config" pod "dcb06ecb-8200-42eb-979e-d407d18b8f10" (UID: "dcb06ecb-8200-42eb-979e-d407d18b8f10"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:45:18.728630 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728591    5845 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728630 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728624    5845 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728630 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728638    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b67s\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-kube-api-access-2b67s\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728649    5845 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-kube-rbac-proxy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728659    5845 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728669    5845 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728678    5845 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-metrics-client-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728687    5845 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-web-config\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728696    5845 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728705    5845 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-config-out\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728714    5845 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728725    5845 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-config\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728733    5845 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-configmap-metrics-client-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728742    5845 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728750    5845 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-db\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728759    5845 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dcb06ecb-8200-42eb-979e-d407d18b8f10-secret-grpc-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728768    5845 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728777    5845 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dcb06ecb-8200-42eb-979e-d407d18b8f10-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:18.728852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:18.728785    5845 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dcb06ecb-8200-42eb-979e-d407d18b8f10-tls-assets\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:45:19.013865 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013831    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229" exitCode=0
Nov 01 21:45:19.013865 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013859    5845 generic.go:356] "Generic (PLEG): container finished" podID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerID="b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5" exitCode=0
Nov 01 21:45:19.014347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013903    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"}
Nov 01 21:45:19.014347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013926    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.014347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013962    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"}
Nov 01 21:45:19.014347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.013993    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dcb06ecb-8200-42eb-979e-d407d18b8f10","Type":"ContainerDied","Data":"1a07b8108dee1c60e1aeb5d8c4ef5c1d318735e50b5346a313f79082f56c0550"}
Nov 01 21:45:19.014347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.014023    5845 scope.go:122] "RemoveContainer" containerID="fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"
Nov 01 21:45:19.021448 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.021431    5845 scope.go:122] "RemoveContainer" containerID="3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"
Nov 01 21:45:19.028048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.028023    5845 scope.go:122] "RemoveContainer" containerID="b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"
Nov 01 21:45:19.034281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.034266    5845 scope.go:122] "RemoveContainer" containerID="183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"
Nov 01 21:45:19.037402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.037377    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:19.041611 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.041588    5845 scope.go:122] "RemoveContainer" containerID="54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"
Nov 01 21:45:19.042266 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.042248    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:19.047931 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.047915    5845 scope.go:122] "RemoveContainer" containerID="76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"
Nov 01 21:45:19.054467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.054451    5845 scope.go:122] "RemoveContainer" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"
Nov 01 21:45:19.060768 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.060748    5845 scope.go:122] "RemoveContainer" containerID="fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"
Nov 01 21:45:19.061051 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.061034    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229\": container with ID starting with fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229 not found: ID does not exist" containerID="fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"
Nov 01 21:45:19.061100 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061059    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"} err="failed to get container status \"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229\": rpc error: code = NotFound desc = could not find container \"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229\": container with ID starting with fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229 not found: ID does not exist"
Nov 01 21:45:19.061100 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061095    5845 scope.go:122] "RemoveContainer" containerID="3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"
Nov 01 21:45:19.061335 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.061315    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19\": container with ID starting with 3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19 not found: ID does not exist" containerID="3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"
Nov 01 21:45:19.061426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061341    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"} err="failed to get container status \"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19\": rpc error: code = NotFound desc = could not find container \"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19\": container with ID starting with 3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19 not found: ID does not exist"
Nov 01 21:45:19.061426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061356    5845 scope.go:122] "RemoveContainer" containerID="b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"
Nov 01 21:45:19.061621 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.061606    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5\": container with ID starting with b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5 not found: ID does not exist" containerID="b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"
Nov 01 21:45:19.061665 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061623    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"} err="failed to get container status \"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5\": rpc error: code = NotFound desc = could not find container \"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5\": container with ID starting with b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5 not found: ID does not exist"
Nov 01 21:45:19.061665 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061636    5845 scope.go:122] "RemoveContainer" containerID="183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"
Nov 01 21:45:19.061860 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.061840    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0\": container with ID starting with 183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0 not found: ID does not exist" containerID="183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"
Nov 01 21:45:19.061906 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061865    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"} err="failed to get container status \"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0\": rpc error: code = NotFound desc = could not find container \"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0\": container with ID starting with 183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0 not found: ID does not exist"
Nov 01 21:45:19.061906 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.061880    5845 scope.go:122] "RemoveContainer" containerID="54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"
Nov 01 21:45:19.062110 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.062095    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1\": container with ID starting with 54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1 not found: ID does not exist" containerID="54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"
Nov 01 21:45:19.062157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062112    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"} err="failed to get container status \"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1\": rpc error: code = NotFound desc = could not find container \"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1\": container with ID starting with 54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1 not found: ID does not exist"
Nov 01 21:45:19.062157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062124    5845 scope.go:122] "RemoveContainer" containerID="76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"
Nov 01 21:45:19.062355 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.062339    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01\": container with ID starting with 76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01 not found: ID does not exist" containerID="76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"
Nov 01 21:45:19.062410 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062356    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"} err="failed to get container status \"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01\": rpc error: code = NotFound desc = could not find container \"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01\": container with ID starting with 76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01 not found: ID does not exist"
Nov 01 21:45:19.062410 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062368    5845 scope.go:122] "RemoveContainer" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"
Nov 01 21:45:19.062578 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:19.062564    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1\": container with ID starting with efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1 not found: ID does not exist" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"
Nov 01 21:45:19.062617 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062581    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"} err="failed to get container status \"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1\": rpc error: code = NotFound desc = could not find container \"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1\": container with ID starting with efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1 not found: ID does not exist"
Nov 01 21:45:19.062617 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062595    5845 scope.go:122] "RemoveContainer" containerID="fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"
Nov 01 21:45:19.062778 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062763    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229"} err="failed to get container status \"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229\": rpc error: code = NotFound desc = could not find container \"fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229\": container with ID starting with fa1d842337b603f3cec9a79bd6c83599a9a26cb015dc70e8f874478bbd404229 not found: ID does not exist"
Nov 01 21:45:19.062821 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062778    5845 scope.go:122] "RemoveContainer" containerID="3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"
Nov 01 21:45:19.062952 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062936    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19"} err="failed to get container status \"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19\": rpc error: code = NotFound desc = could not find container \"3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19\": container with ID starting with 3eeb11b28952282575a47b983f60775a16964f0c9be838ac9f45055d47dc8a19 not found: ID does not exist"
Nov 01 21:45:19.062988 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.062952    5845 scope.go:122] "RemoveContainer" containerID="b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"
Nov 01 21:45:19.063113 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063097    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5"} err="failed to get container status \"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5\": rpc error: code = NotFound desc = could not find container \"b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5\": container with ID starting with b65800ba6b162b66113b47475a6f704c81e25a9bfb66d6c9af15b96c3162eee5 not found: ID does not exist"
Nov 01 21:45:19.063148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063112    5845 scope.go:122] "RemoveContainer" containerID="183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"
Nov 01 21:45:19.063330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063314    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0"} err="failed to get container status \"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0\": rpc error: code = NotFound desc = could not find container \"183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0\": container with ID starting with 183bd60daed7af5d609cd5220b0f4f8f0816386d5b2519b79335d35da21837e0 not found: ID does not exist"
Nov 01 21:45:19.063330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063329    5845 scope.go:122] "RemoveContainer" containerID="54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"
Nov 01 21:45:19.063566 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063538    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1"} err="failed to get container status \"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1\": rpc error: code = NotFound desc = could not find container \"54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1\": container with ID starting with 54abd955da4716becb14cb3fda6ff59ac0d7341510cb30b5acca557edaeab4e1 not found: ID does not exist"
Nov 01 21:45:19.063623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063568    5845 scope.go:122] "RemoveContainer" containerID="76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"
Nov 01 21:45:19.063801 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063785    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01"} err="failed to get container status \"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01\": rpc error: code = NotFound desc = could not find container \"76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01\": container with ID starting with 76248b5510c72dd68b52150b7ff5b8d9763853c8a67d86a683f2d10fc6b61a01 not found: ID does not exist"
Nov 01 21:45:19.063840 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063801    5845 scope.go:122] "RemoveContainer" containerID="efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"
Nov 01 21:45:19.063983 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.063969    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1"} err="failed to get container status \"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1\": rpc error: code = NotFound desc = could not find container \"efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1\": container with ID starting with efb103294d1520f82de6d8bdbb7f0dfb16d5693d27c430dedaae74dd016da8e1 not found: ID does not exist"
Nov 01 21:45:19.074533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074514    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:19.074816 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074803    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="prometheus"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074818    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="prometheus"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074834    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-web"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074839    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-web"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074849    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-thanos"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074854    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-thanos"
Nov 01 21:45:19.074861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074861    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="init-config-reloader"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074865    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="init-config-reloader"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074873    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="thanos-sidecar"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074878    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="thanos-sidecar"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074891    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074896    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074904    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="config-reloader"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074908    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="config-reloader"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074948    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="thanos-sidecar"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074958    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074964    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="config-reloader"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074969    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="prometheus"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074977    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-web"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074983    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="kube-rbac-proxy-thanos"
Nov 01 21:45:19.075060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.074988    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" containerName="init-config-reloader"
Nov 01 21:45:19.078267 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.078254    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.082663 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.082646    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 21:45:19.082927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.082906    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 21:45:19.082927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.082925    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 21:45:19.083071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.082971    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 21:45:19.083185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083171    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 21:45:19.083526 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083506    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 21:45:19.083618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083551    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 21:45:19.083618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083570    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xhnwq\""
Nov 01 21:45:19.083618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083582    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 21:45:19.083747 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083635    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 21:45:19.083747 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.083726    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 21:45:19.085035 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.085015    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 21:45:19.085103 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.085069    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 21:45:19.085152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.085139    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-chkoj1b7amdmk\""
Nov 01 21:45:19.090211 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.090192    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 21:45:19.090278 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.090263    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 21:45:19.098531 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.098511    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:19.232128 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232101    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232265 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232135    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232265 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232155    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232265 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232178    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232339    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232392    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232466 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232414    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232466 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232430    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232466 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232444    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232549 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232513    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232549 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232542    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232612 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232561    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232612 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232576    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232666 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232610    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232699 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232671    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232699 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232688    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232759 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232707    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232759 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232731    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232818 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232768    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.232818 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.232794    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5n6\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-kube-api-access-qm5n6\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333607 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333575    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333609    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5n6\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-kube-api-access-qm5n6\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333644    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333669    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333692    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333726    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.333764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333758    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333783    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333818    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333842    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333868    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333901    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333952    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.333988    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334012    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334059 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334035    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334065    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334090    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334121    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334150    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.334541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334547    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.334887    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.335352    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.335414    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.335519    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.336024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.335723    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.338887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338513    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6db8848b-ac26-44f3-855f-a5217ee57d81-config-out\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.338887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338536    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.338887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338653    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.338887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338719    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.338887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338829    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339128 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338919    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339128 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.338943    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339239 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.339219    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.339280    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6db8848b-ac26-44f3-855f-a5217ee57d81-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339359 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.339321    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.339681 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.339666    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-web-config\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.340338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.340321    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6db8848b-ac26-44f3-855f-a5217ee57d81-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.345763 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.345733    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5n6\" (UniqueName: \"kubernetes.io/projected/6db8848b-ac26-44f3-855f-a5217ee57d81-kube-api-access-qm5n6\") pod \"prometheus-k8s-0\" (UID: \"6db8848b-ac26-44f3-855f-a5217ee57d81\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.396793 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.396767    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:19.686330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:19.685363    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 21:45:20.020580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:20.020543    5845 generic.go:356] "Generic (PLEG): container finished" podID="6db8848b-ac26-44f3-855f-a5217ee57d81" containerID="59dea0475344ea1eb069bf03d9a9f1d5b96c22c71aef60066906c54822b950ed" exitCode=0
Nov 01 21:45:20.020950 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:20.020637    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerDied","Data":"59dea0475344ea1eb069bf03d9a9f1d5b96c22c71aef60066906c54822b950ed"}
Nov 01 21:45:20.020950 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:20.020709    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"99ad054c75ab80a2ea6faffe9a009e72df77d9244bc222dd2e5764fbf0a4d28f"}
Nov 01 21:45:20.858596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:20.858565    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb06ecb-8200-42eb-979e-d407d18b8f10" path="/var/lib/kubelet/pods/dcb06ecb-8200-42eb-979e-d407d18b8f10/volumes"
Nov 01 21:45:21.029764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029725    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"76ca3a3f0303acd859ec5ace3f05457ab3a634a1d79813033a23214ab99bed42"}
Nov 01 21:45:21.030124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029790    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"d7dde3155b2088f4eaa21c849681fc55bafe682982fb80987b32f960d0d8f62a"}
Nov 01 21:45:21.030124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029819    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"fd67cf50543964b5e582a43c62b132b4e183db4d07e49018d7c4858c204d5c27"}
Nov 01 21:45:21.030124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029847    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"dddbdbcd8a68fc1058f5126bea4c477c8a28f07e60b22e85a5ed1ff2e48e6e85"}
Nov 01 21:45:21.030124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029873    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"825d79b0da704fb37e85bc520f6bbdbaef2a6fcb40f7e7a2799463c7ec72fd53"}
Nov 01 21:45:21.030124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.029895    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6db8848b-ac26-44f3-855f-a5217ee57d81","Type":"ContainerStarted","Data":"3ca757fe5debb2b77428927d5270acaa1e79b9055c7cbcd1b04cd6adc325ab30"}
Nov 01 21:45:21.068450 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:21.068038    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.053361328 podStartE2EDuration="2.068018303s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.656975ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:45:21.066381309 +0000 UTC m=+24.322561074" watchObservedRunningTime="2026-11-01 21:45:21.068018303 +0000 UTC m=+24.324198068"
Nov 01 21:45:24.397830 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:24.397800    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:45:33.937546 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:45:33.937514    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f5s44"
Nov 01 21:45:34.813913 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:45:34.813876    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:46:00.769177 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:46:00.769143    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:46:00.784588 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:46:00.784564    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:46:01.201074 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:46:01.200996    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 21:46:04.814700 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:46:04.814665    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:46:34.813653 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:46:34.813616    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:47:04.814632 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:47:04.814596    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:47:34.814119 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:47:34.814082    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:48:04.814526 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:48:04.814490    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:48:07.590110 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.590074    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"]
Nov 01 21:48:07.593490 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.593472    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.598310 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.598277    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 21:48:07.598412 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.598285    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 21:48:07.598412 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.598356    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-nmgwg\""
Nov 01 21:48:07.603126 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.603102    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"]
Nov 01 21:48:07.730121 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.730092    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98lc\" (UniqueName: \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.730284 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.730145    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.730284 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.730264    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.830570 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.830539    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.830701 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.830587    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98lc\" (UniqueName: \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.830701 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.830640    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.830930 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.830909    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.831047 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.831022    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.842889 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.842834    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98lc\" (UniqueName: \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:07.908971 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:07.908917    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:08.180858 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:08.180787    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"]
Nov 01 21:48:08.183694 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:48:08.183657    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a95f493_b4e1_4433_bed7_5be698f854e1.slice/crio-0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409 WatchSource:0}: Error finding container 0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409: Status 404 returned error can't find the container with id 0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409
Nov 01 21:48:08.185635 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:08.185616    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:48:08.624929 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:08.624891    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerStarted","Data":"0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409"}
Nov 01 21:48:13.645222 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:13.645187    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerStarted","Data":"6e7ca851a1fdf49164bdd7ba437f4b25658b6d2707a0f031873f3f76e6200f97"}
Nov 01 21:48:14.649991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:14.649955    5845 generic.go:356] "Generic (PLEG): container finished" podID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerID="6e7ca851a1fdf49164bdd7ba437f4b25658b6d2707a0f031873f3f76e6200f97" exitCode=0
Nov 01 21:48:14.650514 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:14.650015    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerDied","Data":"6e7ca851a1fdf49164bdd7ba437f4b25658b6d2707a0f031873f3f76e6200f97"}
Nov 01 21:48:17.668819 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:17.668783    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerDied","Data":"ab81e9c3becb1c8183d6a40fbd8390a9f7e92ae9dc4182d42dcef2e232a8358b"}
Nov 01 21:48:17.669235 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:17.668779    5845 generic.go:356] "Generic (PLEG): container finished" podID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerID="ab81e9c3becb1c8183d6a40fbd8390a9f7e92ae9dc4182d42dcef2e232a8358b" exitCode=0
Nov 01 21:48:23.701537 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:23.701464    5845 generic.go:356] "Generic (PLEG): container finished" podID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerID="c2a15f36345d651cfa2bf210a6e7c7f4c6e93345fad5c2b6a9a49722e428827f" exitCode=0
Nov 01 21:48:23.701872 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:23.701544    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerDied","Data":"c2a15f36345d651cfa2bf210a6e7c7f4c6e93345fad5c2b6a9a49722e428827f"}
Nov 01 21:48:24.823255 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.823231    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:24.980841 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.980821    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\" (UniqueName: \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\") pod \"7a95f493-b4e1-4433-bed7-5be698f854e1\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") "
Nov 01 21:48:24.980961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.980877    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\") pod \"7a95f493-b4e1-4433-bed7-5be698f854e1\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") "
Nov 01 21:48:24.980961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.980930    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\") pod \"7a95f493-b4e1-4433-bed7-5be698f854e1\" (UID: \"7a95f493-b4e1-4433-bed7-5be698f854e1\") "
Nov 01 21:48:24.981312 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.981262    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle" pod "7a95f493-b4e1-4433-bed7-5be698f854e1" (UID: "7a95f493-b4e1-4433-bed7-5be698f854e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:48:24.982955 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.982932    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc" pod "7a95f493-b4e1-4433-bed7-5be698f854e1" (UID: "7a95f493-b4e1-4433-bed7-5be698f854e1"). InnerVolumeSpecName "kube-api-access-r98lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:48:24.985110 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:24.985084    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util" pod "7a95f493-b4e1-4433-bed7-5be698f854e1" (UID: "7a95f493-b4e1-4433-bed7-5be698f854e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:48:25.082152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.082122    5845 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-bundle\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:48:25.082152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.082148    5845 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a95f493-b4e1-4433-bed7-5be698f854e1-util\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:48:25.082331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.082158    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98lc\" (UniqueName: \"kubernetes.io/projected/7a95f493-b4e1-4433-bed7-5be698f854e1-kube-api-access-r98lc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:48:25.711281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.711244    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm" event={"ID":"7a95f493-b4e1-4433-bed7-5be698f854e1","Type":"ContainerDied","Data":"0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409"}
Nov 01 21:48:25.711581 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.711322    5845 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0111a7c96c03834de6a934c60692ecbcc988f2cdf683437038f2259e6d48f409"
Nov 01 21:48:25.711581 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:25.711256    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447f8fm"
Nov 01 21:48:29.804658 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804625    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"]
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804926    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="pull"
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804936    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="pull"
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804951    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="extract"
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804956    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="extract"
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804967    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="util"
Nov 01 21:48:29.805012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.804972    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="util"
Nov 01 21:48:29.805205 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.805027    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a95f493-b4e1-4433-bed7-5be698f854e1" containerName="extract"
Nov 01 21:48:29.808083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.808064    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:29.826028 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.825993    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-5n478\""
Nov 01 21:48:29.826028 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.826013    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 21:48:29.827716 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.827697    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 21:48:29.828885 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.828862    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"]
Nov 01 21:48:29.922639 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.922610    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e218f0e-148b-42e6-9dd9-cc39297250fa-tmp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:29.922786 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:29.922721    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ghp\" (UniqueName: \"kubernetes.io/projected/7e218f0e-148b-42e6-9dd9-cc39297250fa-kube-api-access-g6ghp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.024142 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.024111    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ghp\" (UniqueName: \"kubernetes.io/projected/7e218f0e-148b-42e6-9dd9-cc39297250fa-kube-api-access-g6ghp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.024327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.024183    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e218f0e-148b-42e6-9dd9-cc39297250fa-tmp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.024919 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.024900    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e218f0e-148b-42e6-9dd9-cc39297250fa-tmp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.036545 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.036523    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ghp\" (UniqueName: \"kubernetes.io/projected/7e218f0e-148b-42e6-9dd9-cc39297250fa-kube-api-access-g6ghp\") pod \"jobset-operator-674cd59dc5-pnl4h\" (UID: \"7e218f0e-148b-42e6-9dd9-cc39297250fa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.134119 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.134061    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"
Nov 01 21:48:30.408327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.408237    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h"]
Nov 01 21:48:30.411627 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:48:30.411597    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e218f0e_148b_42e6_9dd9_cc39297250fa.slice/crio-96ec38a50857590c09d3f6405a9c20cc58b030a4911f80de1341612704ffbf78 WatchSource:0}: Error finding container 96ec38a50857590c09d3f6405a9c20cc58b030a4911f80de1341612704ffbf78: Status 404 returned error can't find the container with id 96ec38a50857590c09d3f6405a9c20cc58b030a4911f80de1341612704ffbf78
Nov 01 21:48:30.730043 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:30.730013    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h" event={"ID":"7e218f0e-148b-42e6-9dd9-cc39297250fa","Type":"ContainerStarted","Data":"96ec38a50857590c09d3f6405a9c20cc58b030a4911f80de1341612704ffbf78"}
Nov 01 21:48:32.741148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:32.741105    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h" event={"ID":"7e218f0e-148b-42e6-9dd9-cc39297250fa","Type":"ContainerStarted","Data":"590457ae927a76f093ee9196debb4163593aa806e71efcca1fac24bfecde8571"}
Nov 01 21:48:32.760160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:32.760113    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pnl4h" podStartSLOduration=1.579064933 podStartE2EDuration="3.760100963s" totalImagesPullingTime="2.18103603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:32.759349708 +0000 UTC m=+216.015529473" watchObservedRunningTime="2026-11-01 21:48:32.760100963 +0000 UTC m=+216.016280718"
Nov 01 21:48:34.813892 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:48:34.813852    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:48:39.754449 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.754413    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"]
Nov 01 21:48:39.759802 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.759775    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.771816 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.771777    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 21:48:39.772097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.771881    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 21:48:39.772097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.771920    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 21:48:39.772097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.771932    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 21:48:39.772097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.771981    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 21:48:39.773264 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.773248    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wbcjm\""
Nov 01 21:48:39.786948 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.785954    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"]
Nov 01 21:48:39.797782 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.797754    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.797897 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.797828    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdv9b\" (UniqueName: \"kubernetes.io/projected/60df8928-8bda-496d-a064-429629ed2583-kube-api-access-fdv9b\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.797965 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.797915    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.798012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.797974    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.798104 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.798082    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.798165 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.798123    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899509 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899517    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899568    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899647    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdv9b\" (UniqueName: \"kubernetes.io/projected/60df8928-8bda-496d-a064-429629ed2583-kube-api-access-fdv9b\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899685    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899964 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899932    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-manifests\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.899964 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.899943    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.902047 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.902018    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60df8928-8bda-496d-a064-429629ed2583-tmp\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.902146 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.902094    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-metrics-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.902279 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.902256    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-webhook-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.902343 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.902322    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60df8928-8bda-496d-a064-429629ed2583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:39.912005 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:39.911984    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdv9b\" (UniqueName: \"kubernetes.io/projected/60df8928-8bda-496d-a064-429629ed2583-kube-api-access-fdv9b\") pod \"opendatahub-operator-controller-manager-98c48f84c-jd65m\" (UID: \"60df8928-8bda-496d-a064-429629ed2583\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:40.079466 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:40.079392    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:40.375546 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:40.375273    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"]
Nov 01 21:48:40.378120 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:48:40.378095    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60df8928_8bda_496d_a064_429629ed2583.slice/crio-19600ff23acb0c825ed5bf4a1c6fa776577f0b850fc4ef0dd1b381d163976e6a WatchSource:0}: Error finding container 19600ff23acb0c825ed5bf4a1c6fa776577f0b850fc4ef0dd1b381d163976e6a: Status 404 returned error can't find the container with id 19600ff23acb0c825ed5bf4a1c6fa776577f0b850fc4ef0dd1b381d163976e6a
Nov 01 21:48:40.773754 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:40.773714    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" event={"ID":"60df8928-8bda-496d-a064-429629ed2583","Type":"ContainerStarted","Data":"19600ff23acb0c825ed5bf4a1c6fa776577f0b850fc4ef0dd1b381d163976e6a"}
Nov 01 21:48:43.787814 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:43.787780    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" event={"ID":"60df8928-8bda-496d-a064-429629ed2583","Type":"ContainerStarted","Data":"c9675795354f9408ddba3615d84e386f7aff61073c8c4dc72881a25ef0885700"}
Nov 01 21:48:44.792997 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:44.792965    5845 generic.go:356] "Generic (PLEG): container finished" podID="60df8928-8bda-496d-a064-429629ed2583" containerID="c9675795354f9408ddba3615d84e386f7aff61073c8c4dc72881a25ef0885700" exitCode=0
Nov 01 21:48:44.793395 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:44.793051    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" event={"ID":"60df8928-8bda-496d-a064-429629ed2583","Type":"ContainerDied","Data":"c9675795354f9408ddba3615d84e386f7aff61073c8c4dc72881a25ef0885700"}
Nov 01 21:48:45.799026 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:45.798986    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" event={"ID":"60df8928-8bda-496d-a064-429629ed2583","Type":"ContainerStarted","Data":"bd18555e8e131435855cb3added78d9b543faf36e70bf4042aef18fdf4e15a4a"}
Nov 01 21:48:45.801361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:45.799207    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:45.845532 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:45.845484    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podStartSLOduration=2.656143289 podStartE2EDuration="6.845471856s" totalImagesPullingTime="3.702846598s" totalInitContainerRuntime="486.481969ms" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:45.843477805 +0000 UTC m=+229.099657569" watchObservedRunningTime="2026-11-01 21:48:45.845471856 +0000 UTC m=+229.101651620"
Nov 01 21:48:53.749965 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.749928    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"]
Nov 01 21:48:53.758449 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.758428    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.769703 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.769682    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rrz8b\""
Nov 01 21:48:53.770875 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.770858    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 21:48:53.770964 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.770947    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 21:48:53.773747 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.773727    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"]
Nov 01 21:48:53.815054 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.815025    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.815167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.815098    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcp8\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-kube-api-access-xdcp8\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.916179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.916151    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.916336 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.916206    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcp8\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-kube-api-access-xdcp8\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.928546 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.928520    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:53.928664 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:53.928644    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcp8\" (UniqueName: \"kubernetes.io/projected/175333af-31b9-45da-b09e-4538fb6a834f-kube-api-access-xdcp8\") pod \"cert-manager-cainjector-5c57bbd4fc-blzdb\" (UID: \"175333af-31b9-45da-b09e-4538fb6a834f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:54.070121 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:54.070049    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"
Nov 01 21:48:54.345479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:54.345402    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb"]
Nov 01 21:48:54.348074 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:48:54.348045    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod175333af_31b9_45da_b09e_4538fb6a834f.slice/crio-78b439bbf40fc9a7ded1fe4c514a0c95c73e02f7acbd66310de2968fcf15bd66 WatchSource:0}: Error finding container 78b439bbf40fc9a7ded1fe4c514a0c95c73e02f7acbd66310de2968fcf15bd66: Status 404 returned error can't find the container with id 78b439bbf40fc9a7ded1fe4c514a0c95c73e02f7acbd66310de2968fcf15bd66
Nov 01 21:48:54.839858 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:54.839829    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb" event={"ID":"175333af-31b9-45da-b09e-4538fb6a834f","Type":"ContainerStarted","Data":"78b439bbf40fc9a7ded1fe4c514a0c95c73e02f7acbd66310de2968fcf15bd66"}
Nov 01 21:48:56.806736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:56.806706    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 21:48:58.373822 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.373791    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"]
Nov 01 21:48:58.375989 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.375968    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.386440 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.386416    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k2z4w\""
Nov 01 21:48:58.387479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.387456    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"]
Nov 01 21:48:58.452929 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.452842    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.452929 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.452913    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6km\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-kube-api-access-hh6km\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.553726 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.553683    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.553890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.553747    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6km\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-kube-api-access-hh6km\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.583709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.583674    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.583916 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.583896    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6km\" (UniqueName: \"kubernetes.io/projected/1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912-kube-api-access-hh6km\") pod \"cert-manager-webhook-5dbbbf75cc-x99jv\" (UID: \"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.688798 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.688438    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:58.858734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.858699    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb" event={"ID":"175333af-31b9-45da-b09e-4538fb6a834f","Type":"ContainerStarted","Data":"9f6f9007e085de674201bb23d9241b9999fb1a308802812470d98fccf083d8f0"}
Nov 01 21:48:58.876965 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.876918    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-blzdb" podStartSLOduration=2.038769092 podStartE2EDuration="5.876904813s" totalImagesPullingTime="3.838135721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:58.876380665 +0000 UTC m=+242.132560428" watchObservedRunningTime="2026-11-01 21:48:58.876904813 +0000 UTC m=+242.133084576"
Nov 01 21:48:58.998588 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:58.998521    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"]
Nov 01 21:48:59.001321 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:48:59.001278    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eb30d9f_8aff_41d9_aa28_9fe8fc8e8912.slice/crio-fd0b3c7fd76754a95a047e53716093eb4ac85a2859bd6102f65c72e1678cdfb6 WatchSource:0}: Error finding container fd0b3c7fd76754a95a047e53716093eb4ac85a2859bd6102f65c72e1678cdfb6: Status 404 returned error can't find the container with id fd0b3c7fd76754a95a047e53716093eb4ac85a2859bd6102f65c72e1678cdfb6
Nov 01 21:48:59.864075 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:59.864038    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" event={"ID":"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912","Type":"ContainerStarted","Data":"5d5cddcb7dc530e8b54523cbf7a6715a71b15dd9718e25c71a61eaa1f0aadcad"}
Nov 01 21:48:59.864585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:59.864210    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:48:59.864585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:59.864240    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" event={"ID":"1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912","Type":"ContainerStarted","Data":"fd0b3c7fd76754a95a047e53716093eb4ac85a2859bd6102f65c72e1678cdfb6"}
Nov 01 21:48:59.891492 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:48:59.891441    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podStartSLOduration=1.891430834 podStartE2EDuration="1.891430834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:48:59.890222647 +0000 UTC m=+243.146402412" watchObservedRunningTime="2026-11-01 21:48:59.891430834 +0000 UTC m=+243.147610593"
Nov 01 21:49:01.006164 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.006125    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbrm6"]
Nov 01 21:49:01.008385 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.008365    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.017306 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.017278    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k8jmk\""
Nov 01 21:49:01.021682 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.021659    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbrm6"]
Nov 01 21:49:01.075706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.075668    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-bound-sa-token\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.075834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.075794    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgc6k\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-kube-api-access-rgc6k\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.177081 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.177055    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgc6k\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-kube-api-access-rgc6k\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.177225 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.177102    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-bound-sa-token\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.195035 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.195012    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgc6k\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-kube-api-access-rgc6k\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.195260 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.195241    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/726e6b52-8dbd-4f88-9535-77748f546abb-bound-sa-token\") pod \"cert-manager-597b95cb84-tbrm6\" (UID: \"726e6b52-8dbd-4f88-9535-77748f546abb\") " pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.318520 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.318440    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tbrm6"
Nov 01 21:49:01.647921 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.647827    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbrm6"]
Nov 01 21:49:01.650508 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:01.650478    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod726e6b52_8dbd_4f88_9535_77748f546abb.slice/crio-15e3f29be97ba3b94f8dc54e424d8a5921e84a55daa1a0d37b005897ab406112 WatchSource:0}: Error finding container 15e3f29be97ba3b94f8dc54e424d8a5921e84a55daa1a0d37b005897ab406112: Status 404 returned error can't find the container with id 15e3f29be97ba3b94f8dc54e424d8a5921e84a55daa1a0d37b005897ab406112
Nov 01 21:49:01.879008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.878974    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tbrm6" event={"ID":"726e6b52-8dbd-4f88-9535-77748f546abb","Type":"ContainerStarted","Data":"dd8d72bd1b88aeb06ac0807a292b94908c6c1a31d18d91203c7cec6f89a26414"}
Nov 01 21:49:01.879008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.879015    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tbrm6" event={"ID":"726e6b52-8dbd-4f88-9535-77748f546abb","Type":"ContainerStarted","Data":"15e3f29be97ba3b94f8dc54e424d8a5921e84a55daa1a0d37b005897ab406112"}
Nov 01 21:49:01.897462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:01.897406    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tbrm6" podStartSLOduration=1.897389539 podStartE2EDuration="1.897389539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:01.897052515 +0000 UTC m=+245.153232280" watchObservedRunningTime="2026-11-01 21:49:01.897389539 +0000 UTC m=+245.153569305"
Nov 01 21:49:04.819203 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:49:04.819163    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:49:05.875747 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:05.875713    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 21:49:12.126151 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.126116    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"]
Nov 01 21:49:12.129727 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.129708    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.139562 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.139538    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8nmd\""
Nov 01 21:49:12.139754 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.139729    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 21:49:12.139842 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.139798    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 21:49:12.140224 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.140191    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 21:49:12.140341 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.140266    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 21:49:12.141010 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.140989    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"]
Nov 01 21:49:12.268324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.268273    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tmp\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.268324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.268315    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.268567 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.268432    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4492\" (UniqueName: \"kubernetes.io/projected/ed7e30f6-adec-4f93-a098-2ab4f61f937d-kube-api-access-v4492\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.369024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.368993    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tmp\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.369024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.369023    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.369208 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.369074    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4492\" (UniqueName: \"kubernetes.io/projected/ed7e30f6-adec-4f93-a098-2ab4f61f937d-kube-api-access-v4492\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.369249 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:49:12.369206    5845 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Nov 01 21:49:12.369317 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:49:12.369288    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs podName:ed7e30f6-adec-4f93-a098-2ab4f61f937d nodeName:}" failed. No retries permitted until 2026-11-01 21:49:12.869268793 +0000 UTC m=+256.125448553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs") pod "kube-auth-proxy-b69dc6f4d-kdssh" (UID: "ed7e30f6-adec-4f93-a098-2ab4f61f937d") : secret "kube-auth-proxy-tls" not found
Nov 01 21:49:12.371131 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.371107    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tmp\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.385012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.384960    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4492\" (UniqueName: \"kubernetes.io/projected/ed7e30f6-adec-4f93-a098-2ab4f61f937d-kube-api-access-v4492\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.873424 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.873392    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:12.875766 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:12.875742    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed7e30f6-adec-4f93-a098-2ab4f61f937d-tls-certs\") pod \"kube-auth-proxy-b69dc6f4d-kdssh\" (UID: \"ed7e30f6-adec-4f93-a098-2ab4f61f937d\") " pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:13.045416 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.045386    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"
Nov 01 21:49:13.316704 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.316679    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh"]
Nov 01 21:49:13.318607 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:13.318575    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7e30f6_adec_4f93_a098_2ab4f61f937d.slice/crio-1fbd189fd6819454e56ef75eb95b9ecd884b5f6d8fc8bb9ec3e76ed8ff05a893 WatchSource:0}: Error finding container 1fbd189fd6819454e56ef75eb95b9ecd884b5f6d8fc8bb9ec3e76ed8ff05a893: Status 404 returned error can't find the container with id 1fbd189fd6819454e56ef75eb95b9ecd884b5f6d8fc8bb9ec3e76ed8ff05a893
Nov 01 21:49:13.811035 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.810998    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"]
Nov 01 21:49:13.854807 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.854767    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"]
Nov 01 21:49:13.854992 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.854860    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:13.858738 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.858712    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 21:49:13.859524 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.859501    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 21:49:13.859524 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.859508    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-t5w4f\""
Nov 01 21:49:13.863525 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.863505    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 21:49:13.908658 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.908632    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:13.928399 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.928377    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:13.928539 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.928470    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:13.933124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.933096    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh" event={"ID":"ed7e30f6-adec-4f93-a098-2ab4f61f937d","Type":"ContainerStarted","Data":"1fbd189fd6819454e56ef75eb95b9ecd884b5f6d8fc8bb9ec3e76ed8ff05a893"}
Nov 01 21:49:13.958975 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.958950    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 21:49:13.959083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.959008    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6fqfs\""
Nov 01 21:49:13.959083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.959032    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 21:49:13.959083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.959040    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 21:49:13.983805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.983781    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1f29aa3-d6a1-4153-b759-0b130813ebf0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:13.983939 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.983829    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:13.984005 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.983933    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:13.984005 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:13.983972    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcfbr\" (UniqueName: \"kubernetes.io/projected/d1f29aa3-d6a1-4153-b759-0b130813ebf0-kube-api-access-fcfbr\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.012567 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.012541    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:14.085417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085335    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.085417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085404    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwn5x\" (UniqueName: \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.085604 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085454    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.085604 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085480    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcfbr\" (UniqueName: \"kubernetes.io/projected/d1f29aa3-d6a1-4153-b759-0b130813ebf0-kube-api-access-fcfbr\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.085687 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085664    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.085735 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085718    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1f29aa3-d6a1-4153-b759-0b130813ebf0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.085782 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085760    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.085958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.085938    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1f29aa3-d6a1-4153-b759-0b130813ebf0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.086805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.086779    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1f29aa3-d6a1-4153-b759-0b130813ebf0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.097143 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.097122    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcfbr\" (UniqueName: \"kubernetes.io/projected/d1f29aa3-d6a1-4153-b759-0b130813ebf0-kube-api-access-fcfbr\") pod \"kserve-module-controller-manager-7566cf5fc8-dz5ct\" (UID: \"d1f29aa3-d6a1-4153-b759-0b130813ebf0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.182753 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.182718    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:14.187255 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.187225    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwn5x\" (UniqueName: \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.187420 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.187291    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.187935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.187895    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.206935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.206907    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwn5x\" (UniqueName: \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\") pod \"limitador-limitador-74dfcfb894-skkbn\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.263741 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.263150    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:14.496415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.496383    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"]
Nov 01 21:49:14.500155 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:14.500127    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f29aa3_d6a1_4153_b759_0b130813ebf0.slice/crio-7a5baf44b82eb92d3ef8046335fcabd03a1e22985398a6e1ba2f26e89eb6dec4 WatchSource:0}: Error finding container 7a5baf44b82eb92d3ef8046335fcabd03a1e22985398a6e1ba2f26e89eb6dec4: Status 404 returned error can't find the container with id 7a5baf44b82eb92d3ef8046335fcabd03a1e22985398a6e1ba2f26e89eb6dec4
Nov 01 21:49:14.567234 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.567206    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:14.570565 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:14.570533    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e08da17_dfef_4e5c_9d58_5fcc77a0dbbf.slice/crio-553f99d36dfeee1286ce08d8da08e9dde143d9642d59f79d12130d9fd8a3f54d WatchSource:0}: Error finding container 553f99d36dfeee1286ce08d8da08e9dde143d9642d59f79d12130d9fd8a3f54d: Status 404 returned error can't find the container with id 553f99d36dfeee1286ce08d8da08e9dde143d9642d59f79d12130d9fd8a3f54d
Nov 01 21:49:14.886264 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.886233    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:49:14.888974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.888949    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:49:14.899324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.899272    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:49:14.934285 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.934256    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmvjn\""
Nov 01 21:49:14.940689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.940662    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" event={"ID":"d1f29aa3-d6a1-4153-b759-0b130813ebf0","Type":"ContainerStarted","Data":"7a5baf44b82eb92d3ef8046335fcabd03a1e22985398a6e1ba2f26e89eb6dec4"}
Nov 01 21:49:14.942351 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.942327    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" event={"ID":"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf","Type":"ContainerStarted","Data":"553f99d36dfeee1286ce08d8da08e9dde143d9642d59f79d12130d9fd8a3f54d"}
Nov 01 21:49:14.995017 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:14.994984    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l62mm\" (UniqueName: \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\") pod \"authorino-56c6c9ddd5-mrztr\" (UID: \"e94fc030-797b-402c-8afb-2e7d9d56605b\") " pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:49:15.096466 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:15.096430    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l62mm\" (UniqueName: \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\") pod \"authorino-56c6c9ddd5-mrztr\" (UID: \"e94fc030-797b-402c-8afb-2e7d9d56605b\") " pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:49:15.110193 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:15.110159    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l62mm\" (UniqueName: \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\") pod \"authorino-56c6c9ddd5-mrztr\" (UID: \"e94fc030-797b-402c-8afb-2e7d9d56605b\") " pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:49:15.205555 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:15.205485    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:49:15.502647 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:15.502610    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:49:15.506638 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:15.506604    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode94fc030_797b_402c_8afb_2e7d9d56605b.slice/crio-a05743007a846c4474b97fce77603ddded382c87aef509d0baf2bd64b89c854e WatchSource:0}: Error finding container a05743007a846c4474b97fce77603ddded382c87aef509d0baf2bd64b89c854e: Status 404 returned error can't find the container with id a05743007a846c4474b97fce77603ddded382c87aef509d0baf2bd64b89c854e
Nov 01 21:49:15.951238 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:15.951139    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" event={"ID":"e94fc030-797b-402c-8afb-2e7d9d56605b","Type":"ContainerStarted","Data":"a05743007a846c4474b97fce77603ddded382c87aef509d0baf2bd64b89c854e"}
Nov 01 21:49:16.960208 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:16.960170    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh" event={"ID":"ed7e30f6-adec-4f93-a098-2ab4f61f937d","Type":"ContainerStarted","Data":"7262320b53e7972c712e2c4ff15e8ba7b9a001368c7446b6402a652a1931c1ae"}
Nov 01 21:49:16.964281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:16.964161    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" event={"ID":"d1f29aa3-d6a1-4153-b759-0b130813ebf0","Type":"ContainerStarted","Data":"c6f054f7c58064d67d1e60250330fe205e80f404724fc452a69844379ebb315a"}
Nov 01 21:49:16.964281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:16.964244    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:16.983090 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:16.983038    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b69dc6f4d-kdssh" podStartSLOduration=1.9177217149999999 podStartE2EDuration="4.983025178s" totalImagesPullingTime="3.065303463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:16.982006249 +0000 UTC m=+260.238186024" watchObservedRunningTime="2026-11-01 21:49:16.983025178 +0000 UTC m=+260.239204940"
Nov 01 21:49:17.971493 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:17.971453    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" event={"ID":"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf","Type":"ContainerStarted","Data":"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"}
Nov 01 21:49:17.972170 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:17.972140    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:17.989968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:17.989927    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" podStartSLOduration=1.706891956 podStartE2EDuration="4.989916215s" totalImagesPullingTime="3.283024259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:17.989474518 +0000 UTC m=+261.245654283" watchObservedRunningTime="2026-11-01 21:49:17.989916215 +0000 UTC m=+261.246095978"
Nov 01 21:49:17.990236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:17.990208    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podStartSLOduration=3.003434877 podStartE2EDuration="4.99019878s" totalImagesPullingTime="1.986763903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:17.005031707 +0000 UTC m=+260.261211470" watchObservedRunningTime="2026-11-01 21:49:17.99019878 +0000 UTC m=+261.246378576"
Nov 01 21:49:18.977152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:18.977112    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" event={"ID":"e94fc030-797b-402c-8afb-2e7d9d56605b","Type":"ContainerStarted","Data":"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"}
Nov 01 21:49:19.039346 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:19.039280    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" podStartSLOduration=1.910276189 podStartE2EDuration="5.039265447s" totalImagesPullingTime="3.128989258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:19.03794581 +0000 UTC m=+262.294125574" watchObservedRunningTime="2026-11-01 21:49:19.039265447 +0000 UTC m=+262.295445209"
Nov 01 21:49:28.090895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.090863    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:28.091309 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.091077    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" podUID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" containerName="limitador" containerID="cri-o://6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752" gracePeriod=30
Nov 01 21:49:28.091778 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.091760    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:28.630910 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.630886    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:28.708409 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.708312    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\" (UniqueName: \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\") pod \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") "
Nov 01 21:49:28.708570 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.708413    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\" (UniqueName: \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\") pod \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\" (UID: \"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf\") "
Nov 01 21:49:28.709273 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.709233    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file" pod "0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" (UID: "0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:49:28.711357 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.711333    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x" pod "0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" (UID: "0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf"). InnerVolumeSpecName "kube-api-access-kwn5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:49:28.809998 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.809961    5845 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-config-file\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:49:28.809998 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:28.809995    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwn5x\" (UniqueName: \"kubernetes.io/projected/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf-kube-api-access-kwn5x\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:49:29.016600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.016566    5845 generic.go:356] "Generic (PLEG): container finished" podID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" containerID="6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752" exitCode=0
Nov 01 21:49:29.016790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.016618    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn"
Nov 01 21:49:29.016790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.016651    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" event={"ID":"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf","Type":"ContainerDied","Data":"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"}
Nov 01 21:49:29.016790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.016706    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-skkbn" event={"ID":"0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf","Type":"ContainerDied","Data":"553f99d36dfeee1286ce08d8da08e9dde143d9642d59f79d12130d9fd8a3f54d"}
Nov 01 21:49:29.016790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.016728    5845 scope.go:122] "RemoveContainer" containerID="6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"
Nov 01 21:49:29.025191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.025176    5845 scope.go:122] "RemoveContainer" containerID="6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"
Nov 01 21:49:29.025453 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:49:29.025435    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752\": container with ID starting with 6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752 not found: ID does not exist" containerID="6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"
Nov 01 21:49:29.025505 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.025463    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752"} err="failed to get container status \"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752\": rpc error: code = NotFound desc = could not find container \"6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752\": container with ID starting with 6370d1b528ccb423c7c168eafbc8bb39f08b0d4fd0741f787901a9e2c8b0d752 not found: ID does not exist"
Nov 01 21:49:29.033333 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.033284    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:29.037712 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:29.037691    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-skkbn"]
Nov 01 21:49:30.859094 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:30.859056    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" path="/var/lib/kubelet/pods/0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf/volumes"
Nov 01 21:49:34.659609 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.659577    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"]
Nov 01 21:49:34.660033 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.659938    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" containerName="limitador"
Nov 01 21:49:34.660033 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.659950    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" containerName="limitador"
Nov 01 21:49:34.660033 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.660014    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e08da17-dfef-4e5c-9d58-5fcc77a0dbbf" containerName="limitador"
Nov 01 21:49:34.666753 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.666731    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.672157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.672131    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"]
Nov 01 21:49:34.673886 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.673499    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 21:49:34.674622 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.674600    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 21:49:34.674716 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.674632    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 21:49:34.674716 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.674609    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-frp7s\""
Nov 01 21:49:34.758286 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.758246    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/533a4e94-3fb6-4bb4-b248-0a256638d509-manager-config\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.758286 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.758292    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-cert\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.758576 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.758406    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47t7\" (UniqueName: \"kubernetes.io/projected/533a4e94-3fb6-4bb4-b248-0a256638d509-kube-api-access-z47t7\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.758576 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.758531    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-metrics-certs\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.818817 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:49:34.818751    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:49:34.859543 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.859514    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-metrics-certs\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.859730 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.859555    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/533a4e94-3fb6-4bb4-b248-0a256638d509-manager-config\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.859730 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.859578    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-cert\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.859730 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.859623    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47t7\" (UniqueName: \"kubernetes.io/projected/533a4e94-3fb6-4bb4-b248-0a256638d509-kube-api-access-z47t7\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.860259 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.860237    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/533a4e94-3fb6-4bb4-b248-0a256638d509-manager-config\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.861961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.861939    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-cert\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.862115 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.862097    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/533a4e94-3fb6-4bb4-b248-0a256638d509-metrics-certs\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.871881 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.871861    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47t7\" (UniqueName: \"kubernetes.io/projected/533a4e94-3fb6-4bb4-b248-0a256638d509-kube-api-access-z47t7\") pod \"jobset-controller-manager-75d8948445-6nwm2\" (UID: \"533a4e94-3fb6-4bb4-b248-0a256638d509\") " pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:34.980223 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:34.980200    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:35.251366 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:35.251338    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"]
Nov 01 21:49:35.253925 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:49:35.253901    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533a4e94_3fb6_4bb4_b248_0a256638d509.slice/crio-646a85e838f43ecffc0b8c8f96ebaacad561833ee980cd99287bf7506009264e WatchSource:0}: Error finding container 646a85e838f43ecffc0b8c8f96ebaacad561833ee980cd99287bf7506009264e: Status 404 returned error can't find the container with id 646a85e838f43ecffc0b8c8f96ebaacad561833ee980cd99287bf7506009264e
Nov 01 21:49:36.050097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:36.050060    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" event={"ID":"533a4e94-3fb6-4bb4-b248-0a256638d509","Type":"ContainerStarted","Data":"646a85e838f43ecffc0b8c8f96ebaacad561833ee980cd99287bf7506009264e"}
Nov 01 21:49:37.978731 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:37.978530    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 21:49:39.066027 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:39.065990    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" event={"ID":"533a4e94-3fb6-4bb4-b248-0a256638d509","Type":"ContainerStarted","Data":"4f9eb305ea653fb700a433376918da577fec784ffa8e9b8ff265e0b29567dd5b"}
Nov 01 21:49:39.066487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:39.066137    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:39.086906 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:39.086856    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podStartSLOduration=1.4669542340000001 podStartE2EDuration="5.086839564s" totalImagesPullingTime="3.61988533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:49:39.086773834 +0000 UTC m=+282.342953598" watchObservedRunningTime="2026-11-01 21:49:39.086839564 +0000 UTC m=+282.343019328"
Nov 01 21:49:50.075378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:50.075342    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 21:49:56.885784 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:56.885757    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:49:56.887781 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:49:56.887761    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:50:04.818774 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:50:04.818729    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:50:34.668705 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.668675    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 21:50:34.677133 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.677112    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.684789 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.684765    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 21:50:34.684789 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.684766    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 21:50:34.684968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.684799    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 21:50:34.684968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.684817    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 21:50:34.684968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.684946    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zltrj\""
Nov 01 21:50:34.686453 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.686430    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 21:50:34.784462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.784428    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.784462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.784466    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.784673 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.784483    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74jq\" (UniqueName: \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.784673 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.784547    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.815121 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:50:34.815090    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:50:34.886054 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.886017    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.886054 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.886057    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.886278 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.886082    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74jq\" (UniqueName: \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.886278 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.886120    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.886854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.886831    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.888458 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.888434    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.888536 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.888490    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.898052 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.898029    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74jq\" (UniqueName: \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\") pod \"maas-api-976bc5795-qwgk4\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") " pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:34.991249 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:34.991222    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:35.264996 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:35.264794    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 21:50:35.267712 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:50:35.267671    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod889f0c02_0248_4f0e_afc9_f3f7730f3aac.slice/crio-8fd827a7548c9c76eb5e261d093b9e9b0a9424e89cec6fb9f9d67f86db228c20 WatchSource:0}: Error finding container 8fd827a7548c9c76eb5e261d093b9e9b0a9424e89cec6fb9f9d67f86db228c20: Status 404 returned error can't find the container with id 8fd827a7548c9c76eb5e261d093b9e9b0a9424e89cec6fb9f9d67f86db228c20
Nov 01 21:50:35.319743 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:35.319709    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" event={"ID":"889f0c02-0248-4f0e-afc9-f3f7730f3aac","Type":"ContainerStarted","Data":"8fd827a7548c9c76eb5e261d093b9e9b0a9424e89cec6fb9f9d67f86db228c20"}
Nov 01 21:50:37.558165 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:37.558126    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:50:37.558663 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:37.558410    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" podUID="e94fc030-797b-402c-8afb-2e7d9d56605b" containerName="authorino" containerID="cri-o://84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7" gracePeriod=30
Nov 01 21:50:37.811218 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:37.811156    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:50:37.913459 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:37.913415    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\" (UniqueName: \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\") pod \"e94fc030-797b-402c-8afb-2e7d9d56605b\" (UID: \"e94fc030-797b-402c-8afb-2e7d9d56605b\") "
Nov 01 21:50:37.915487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:37.915462    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm" pod "e94fc030-797b-402c-8afb-2e7d9d56605b" (UID: "e94fc030-797b-402c-8afb-2e7d9d56605b"). InnerVolumeSpecName "kube-api-access-l62mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:50:38.014689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.014656    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l62mm\" (UniqueName: \"kubernetes.io/projected/e94fc030-797b-402c-8afb-2e7d9d56605b-kube-api-access-l62mm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:50:38.335774 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.335734    5845 generic.go:356] "Generic (PLEG): container finished" podID="e94fc030-797b-402c-8afb-2e7d9d56605b" containerID="84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7" exitCode=0
Nov 01 21:50:38.335943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.335801    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr"
Nov 01 21:50:38.335943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.335820    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" event={"ID":"e94fc030-797b-402c-8afb-2e7d9d56605b","Type":"ContainerDied","Data":"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"}
Nov 01 21:50:38.335943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.335866    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mrztr" event={"ID":"e94fc030-797b-402c-8afb-2e7d9d56605b","Type":"ContainerDied","Data":"a05743007a846c4474b97fce77603ddded382c87aef509d0baf2bd64b89c854e"}
Nov 01 21:50:38.335943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.335888    5845 scope.go:122] "RemoveContainer" containerID="84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"
Nov 01 21:50:38.337597 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.337564    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" event={"ID":"889f0c02-0248-4f0e-afc9-f3f7730f3aac","Type":"ContainerStarted","Data":"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"}
Nov 01 21:50:38.337750 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.337656    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:50:38.345038 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.345021    5845 scope.go:122] "RemoveContainer" containerID="84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"
Nov 01 21:50:38.345292 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:50:38.345275    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7\": container with ID starting with 84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7 not found: ID does not exist" containerID="84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"
Nov 01 21:50:38.345397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.345328    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7"} err="failed to get container status \"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7\": rpc error: code = NotFound desc = could not find container \"84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7\": container with ID starting with 84fb266588f7fcf495dd180d8a820976a7378ac54414b2a4f7d278adbbbabff7 not found: ID does not exist"
Nov 01 21:50:38.361594 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.361551    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" podStartSLOduration=1.9975556079999999 podStartE2EDuration="4.361537479s" totalImagesPullingTime="2.363981871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:50:38.360777807 +0000 UTC m=+341.616957571" watchObservedRunningTime="2026-11-01 21:50:38.361537479 +0000 UTC m=+341.617717239"
Nov 01 21:50:38.375947 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.375924    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:50:38.377682 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.377661    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mrztr"]
Nov 01 21:50:38.859184 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:38.859154    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94fc030-797b-402c-8afb-2e7d9d56605b" path="/var/lib/kubelet/pods/e94fc030-797b-402c-8afb-2e7d9d56605b/volumes"
Nov 01 21:50:44.349136 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:50:44.349106    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 21:51:04.814369 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:51:04.814330    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:51:34.814592 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:51:34.814542    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:52:04.815232 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:52:04.815185    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:52:14.922039 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.922006    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"]
Nov 01 21:52:14.922595 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.922568    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e94fc030-797b-402c-8afb-2e7d9d56605b" containerName="authorino"
Nov 01 21:52:14.922595 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.922588    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94fc030-797b-402c-8afb-2e7d9d56605b" containerName="authorino"
Nov 01 21:52:14.922786 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.922689    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94fc030-797b-402c-8afb-2e7d9d56605b" containerName="authorino"
Nov 01 21:52:14.926114 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.926095    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:14.931496 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.931155    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 21:52:14.931496 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.931223    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rfgwk\""
Nov 01 21:52:14.931496 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.931272    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 01 21:52:14.932705 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.932687    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 21:52:14.935114 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:14.935091    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"]
Nov 01 21:52:15.026619 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026590    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.026761 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026637    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.026761 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026706    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.026761 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026745    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c63df49-6605-4914-9756-a8dcb41d4a15-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.026877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026765    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.026877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.026800    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2ls\" (UniqueName: \"kubernetes.io/projected/2c63df49-6605-4914-9756-a8dcb41d4a15-kube-api-access-2b2ls\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128050 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128021    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128056    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128080    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c63df49-6605-4914-9756-a8dcb41d4a15-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128184    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128235    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2ls\" (UniqueName: \"kubernetes.io/projected/2c63df49-6605-4914-9756-a8dcb41d4a15-kube-api-access-2b2ls\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128326    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128511 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128428    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128571 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128548    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.128653 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.128629    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.130251 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.130229    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c63df49-6605-4914-9756-a8dcb41d4a15-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.130738 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.130719    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c63df49-6605-4914-9756-a8dcb41d4a15-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.140125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.140105    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2ls\" (UniqueName: \"kubernetes.io/projected/2c63df49-6605-4914-9756-a8dcb41d4a15-kube-api-access-2b2ls\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fszvp\" (UID: \"2c63df49-6605-4914-9756-a8dcb41d4a15\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.240487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.240462    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:15.511900 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.511785    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"]
Nov 01 21:52:15.514094 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:15.514068    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c63df49_6605_4914_9756_a8dcb41d4a15.slice/crio-5f9ae77bda703a69f9a53ae8e3f1934c45ff38bd9dc4bcf1b77e7508fcaacafa WatchSource:0}: Error finding container 5f9ae77bda703a69f9a53ae8e3f1934c45ff38bd9dc4bcf1b77e7508fcaacafa: Status 404 returned error can't find the container with id 5f9ae77bda703a69f9a53ae8e3f1934c45ff38bd9dc4bcf1b77e7508fcaacafa
Nov 01 21:52:15.750193 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:15.750156    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" event={"ID":"2c63df49-6605-4914-9756-a8dcb41d4a15","Type":"ContainerStarted","Data":"5f9ae77bda703a69f9a53ae8e3f1934c45ff38bd9dc4bcf1b77e7508fcaacafa"}
Nov 01 21:52:16.072841 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.072806    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"]
Nov 01 21:52:16.078422 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.078397    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.080672 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.080643    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"]
Nov 01 21:52:16.085237 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.085201    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 21:52:16.138824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.138733    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.138824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.138799    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.139052 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.138894    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgvv\" (UniqueName: \"kubernetes.io/projected/0a2d2561-28ba-4f2e-bd41-88b863f73b11-kube-api-access-spgvv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.139052 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.138941    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.139052 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.138968    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.139052 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.139026    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239727 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239691    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239911 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239776    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239911 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239800    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239911 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239819    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgvv\" (UniqueName: \"kubernetes.io/projected/0a2d2561-28ba-4f2e-bd41-88b863f73b11-kube-api-access-spgvv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239911 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239851    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.239911 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.239870    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.240284 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.240260    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.240775 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.240709    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.240775 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.240737    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.242119 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.242096    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d2561-28ba-4f2e-bd41-88b863f73b11-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.242772 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.242753    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d2561-28ba-4f2e-bd41-88b863f73b11-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.253167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.253141    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgvv\" (UniqueName: \"kubernetes.io/projected/0a2d2561-28ba-4f2e-bd41-88b863f73b11-kube-api-access-spgvv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm\" (UID: \"0a2d2561-28ba-4f2e-bd41-88b863f73b11\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.398188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.398095    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:16.684188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.684115    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"]
Nov 01 21:52:16.686776 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:16.686748    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d2561_28ba_4f2e_bd41_88b863f73b11.slice/crio-34ea7893ad1d6def2f7c600a693ec04c0fe25db8f6b613d41eebf640f26b846f WatchSource:0}: Error finding container 34ea7893ad1d6def2f7c600a693ec04c0fe25db8f6b613d41eebf640f26b846f: Status 404 returned error can't find the container with id 34ea7893ad1d6def2f7c600a693ec04c0fe25db8f6b613d41eebf640f26b846f
Nov 01 21:52:16.756360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.756333    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" event={"ID":"0a2d2561-28ba-4f2e-bd41-88b863f73b11","Type":"ContainerStarted","Data":"34ea7893ad1d6def2f7c600a693ec04c0fe25db8f6b613d41eebf640f26b846f"}
Nov 01 21:52:16.758026 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.758006    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" event={"ID":"2c63df49-6605-4914-9756-a8dcb41d4a15","Type":"ContainerStarted","Data":"60925d325e7db1be9aa423e34530e28c30ee17fd0c854fdacafe70b50c25d32a"}
Nov 01 21:52:16.779803 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:16.779752    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podStartSLOduration=1.6031739699999998 podStartE2EDuration="2.779737806s" totalImagesPullingTime="1.176563836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:16.777535289 +0000 UTC m=+440.033715055" watchObservedRunningTime="2026-11-01 21:52:16.779737806 +0000 UTC m=+440.035917571"
Nov 01 21:52:17.241563 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.241526    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:17.659693 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.659654    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cdk5r"]
Nov 01 21:52:17.663281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.663259    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.668185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.668164    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 21:52:17.674945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.674922    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cdk5r"]
Nov 01 21:52:17.753922 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.753896    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-home\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.754053 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.753926    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.754053 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.753948    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27c16ff-fa1f-4815-902c-8a05f5070f07-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.754053 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.754018    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjdt\" (UniqueName: \"kubernetes.io/projected/b27c16ff-fa1f-4815-902c-8a05f5070f07-kube-api-access-ksjdt\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.754195 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.754173    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.754244 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.754214    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.764515 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.764487    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" event={"ID":"0a2d2561-28ba-4f2e-bd41-88b863f73b11","Type":"ContainerStarted","Data":"156357f69f215cf04858247c7f64b096c3249c3d481ce6ef3bc6fa2b6164d77b"}
Nov 01 21:52:17.785361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.785316    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podStartSLOduration=1.477979459 podStartE2EDuration="1.785286891s" totalImagesPullingTime="307.307432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:17.784366877 +0000 UTC m=+441.040546643" watchObservedRunningTime="2026-11-01 21:52:17.785286891 +0000 UTC m=+441.041466653"
Nov 01 21:52:17.855410 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855386    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-home\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855413    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855453    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27c16ff-fa1f-4815-902c-8a05f5070f07-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855479    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjdt\" (UniqueName: \"kubernetes.io/projected/b27c16ff-fa1f-4815-902c-8a05f5070f07-kube-api-access-ksjdt\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855576    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855631    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855876 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855845    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.855962 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855899    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-home\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.856025 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.855997    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.857928 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.857911    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27c16ff-fa1f-4815-902c-8a05f5070f07-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.858216 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.858197    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27c16ff-fa1f-4815-902c-8a05f5070f07-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.868047 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.868025    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjdt\" (UniqueName: \"kubernetes.io/projected/b27c16ff-fa1f-4815-902c-8a05f5070f07-kube-api-access-ksjdt\") pod \"echo-simulator-kserve-64b6f8f96c-cdk5r\" (UID: \"b27c16ff-fa1f-4815-902c-8a05f5070f07\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:17.977462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:17.977437    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:18.248668 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:18.248605    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cdk5r"]
Nov 01 21:52:18.252256 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:18.252227    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27c16ff_fa1f_4815_902c_8a05f5070f07.slice/crio-285dde0e81bc2d453d9458c4682eab8088180f43c8dc2758ce8cd6f1fb3dc267 WatchSource:0}: Error finding container 285dde0e81bc2d453d9458c4682eab8088180f43c8dc2758ce8cd6f1fb3dc267: Status 404 returned error can't find the container with id 285dde0e81bc2d453d9458c4682eab8088180f43c8dc2758ce8cd6f1fb3dc267
Nov 01 21:52:18.399291 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:18.399258    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:18.770000 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:18.769969    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" event={"ID":"b27c16ff-fa1f-4815-902c-8a05f5070f07","Type":"ContainerStarted","Data":"285dde0e81bc2d453d9458c4682eab8088180f43c8dc2758ce8cd6f1fb3dc267"}
Nov 01 21:52:20.781487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:20.781449    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" event={"ID":"b27c16ff-fa1f-4815-902c-8a05f5070f07","Type":"ContainerStarted","Data":"dce192a1f36cc5d0584d1977540f31e8cef26b10d28070c5d0e1a93f3cb7421f"}
Nov 01 21:52:20.804800 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:20.804742    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podStartSLOduration=2.09790206 podStartE2EDuration="3.804704376s" totalImagesPullingTime="1.706802316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:20.803433762 +0000 UTC m=+444.059613547" watchObservedRunningTime="2026-11-01 21:52:20.804704376 +0000 UTC m=+444.060884138"
Nov 01 21:52:20.978309 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:20.978267    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:21.626795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.626762    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"]
Nov 01 21:52:21.631050 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.631030    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.635968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.635947    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 21:52:21.642914 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.642891    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"]
Nov 01 21:52:21.689803 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.689771    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.689966 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.689829    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.690029 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.689982    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.690083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.690026    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.690138 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.690113    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h594\" (UniqueName: \"kubernetes.io/projected/83f9f654-a050-4cdc-bc76-d2ea43698bdd-kube-api-access-2h594\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.690185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.690161    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791240 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791206    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791255    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791311    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791430    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h594\" (UniqueName: \"kubernetes.io/projected/83f9f654-a050-4cdc-bc76-d2ea43698bdd-kube-api-access-2h594\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791509    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791728    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791753    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.791966 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.791930    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.793518 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.793497    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83f9f654-a050-4cdc-bc76-d2ea43698bdd-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.793735 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.793717    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83f9f654-a050-4cdc-bc76-d2ea43698bdd-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.805497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.805475    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h594\" (UniqueName: \"kubernetes.io/projected/83f9f654-a050-4cdc-bc76-d2ea43698bdd-kube-api-access-2h594\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m\" (UID: \"83f9f654-a050-4cdc-bc76-d2ea43698bdd\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:21.942637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:21.942563    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:22.216361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:22.216272    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"]
Nov 01 21:52:22.218952 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:22.218917    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83f9f654_a050_4cdc_bc76_d2ea43698bdd.slice/crio-4c1bb93db327fa922e137626c00b656469ae36f1770aa770d411324798566d5f WatchSource:0}: Error finding container 4c1bb93db327fa922e137626c00b656469ae36f1770aa770d411324798566d5f: Status 404 returned error can't find the container with id 4c1bb93db327fa922e137626c00b656469ae36f1770aa770d411324798566d5f
Nov 01 21:52:22.797355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:22.797255    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" event={"ID":"83f9f654-a050-4cdc-bc76-d2ea43698bdd","Type":"ContainerStarted","Data":"250dc5bbd7853ae4d829d533dfcecb2b232b30fb7fb454e4b97b027a7450aa0e"}
Nov 01 21:52:22.797355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:22.797325    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" event={"ID":"83f9f654-a050-4cdc-bc76-d2ea43698bdd","Type":"ContainerStarted","Data":"4c1bb93db327fa922e137626c00b656469ae36f1770aa770d411324798566d5f"}
Nov 01 21:52:22.819877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:22.819829    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podStartSLOduration=1.523040425 podStartE2EDuration="1.819817352s" totalImagesPullingTime="296.776927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:22.818954556 +0000 UTC m=+446.075134320" watchObservedRunningTime="2026-11-01 21:52:22.819817352 +0000 UTC m=+446.075997114"
Nov 01 21:52:22.942843 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:22.942812    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:25.241210 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:25.241178    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:25.253456 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:25.253436    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:25.824159 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:25.824132    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 21:52:26.399107 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:26.399075    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:26.411141 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:26.411121    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:26.828267 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:26.828240    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 21:52:27.978228 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:27.978199    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:27.990138 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:27.990118    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:28.840071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:28.840041    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 21:52:31.943353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:31.943324    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:31.955642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:31.955618    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:32.865171 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:32.865144    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:52:34.816265 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:52:34.816226    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:52:43.129072 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.128998    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-697bf54cfb-r86kt"]
Nov 01 21:52:43.136817 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.136798    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.140929 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.140904    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-697bf54cfb-r86kt"]
Nov 01 21:52:43.145607 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.145582    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 21:52:43.145607 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.145596    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tmvjn\""
Nov 01 21:52:43.145738 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.145604    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 21:52:43.145738 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.145692    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 21:52:43.191548 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.191524    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st92\" (UniqueName: \"kubernetes.io/projected/12e40591-1504-4ecd-a486-c6607a5c0225-kube-api-access-5st92\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.191700 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.191582    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12e40591-1504-4ecd-a486-c6607a5c0225-tls-cert\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.292095 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.292069    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st92\" (UniqueName: \"kubernetes.io/projected/12e40591-1504-4ecd-a486-c6607a5c0225-kube-api-access-5st92\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.292228 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.292124    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12e40591-1504-4ecd-a486-c6607a5c0225-tls-cert\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.294517 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.294491    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/12e40591-1504-4ecd-a486-c6607a5c0225-tls-cert\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.304205 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.304186    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st92\" (UniqueName: \"kubernetes.io/projected/12e40591-1504-4ecd-a486-c6607a5c0225-kube-api-access-5st92\") pod \"authorino-697bf54cfb-r86kt\" (UID: \"12e40591-1504-4ecd-a486-c6607a5c0225\") " pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.450895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.450836    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-697bf54cfb-r86kt"
Nov 01 21:52:43.721006 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.720976    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-697bf54cfb-r86kt"]
Nov 01 21:52:43.724980 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:43.724955    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e40591_1504_4ecd_a486_c6607a5c0225.slice/crio-6047ff7cc20285a12a94e91c6bda7350d83e834eb24a1c521a18da109b7d850c WatchSource:0}: Error finding container 6047ff7cc20285a12a94e91c6bda7350d83e834eb24a1c521a18da109b7d850c: Status 404 returned error can't find the container with id 6047ff7cc20285a12a94e91c6bda7350d83e834eb24a1c521a18da109b7d850c
Nov 01 21:52:43.902125 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:43.902094    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-697bf54cfb-r86kt" event={"ID":"12e40591-1504-4ecd-a486-c6607a5c0225","Type":"ContainerStarted","Data":"6047ff7cc20285a12a94e91c6bda7350d83e834eb24a1c521a18da109b7d850c"}
Nov 01 21:52:44.909995 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:44.909951    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-697bf54cfb-r86kt" event={"ID":"12e40591-1504-4ecd-a486-c6607a5c0225","Type":"ContainerStarted","Data":"e44befc0f963a98333a8de052096259ed6d9bfe82bf911446e401309790b9967"}
Nov 01 21:52:44.928007 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:44.927959    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-697bf54cfb-r86kt" podStartSLOduration=0.975809274 podStartE2EDuration="1.92794638s" totalImagesPullingTime="952.137106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:44.926954175 +0000 UTC m=+468.183133939" watchObservedRunningTime="2026-11-01 21:52:44.92794638 +0000 UTC m=+468.184126143"
Nov 01 21:52:52.372625 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.372595    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"]
Nov 01 21:52:52.376040 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.376020    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.381369 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.381348    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 21:52:52.390196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.390171    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"]
Nov 01 21:52:52.463122 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.463095    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tfhbq\""
Nov 01 21:52:52.495914 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.495883    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.495921    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a30e41-ed57-4bff-92a2-38160219bfb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.495941    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496176 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496092    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496223 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496167    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496223 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496206    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496344 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496224    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496344 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496243    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.496411 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.496393    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrlf\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-kube-api-access-szrlf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.598395 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.598359    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a30e41-ed57-4bff-92a2-38160219bfb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.598637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.598617    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.598780 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.598766    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.598899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.598886    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.599024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.599011    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.599127 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.599113    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.599230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.599216    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.599417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.599397    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrlf\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-kube-api-access-szrlf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.599580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.599564    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.603711 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.602948    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.603711 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.603597    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a30e41-ed57-4bff-92a2-38160219bfb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.605245 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.605215    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.605369 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.605227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.605897 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.605871    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.605981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.605968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.606058 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.606034    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.615795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.615774    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.616191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.616169    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrlf\" (UniqueName: \"kubernetes.io/projected/73a30e41-ed57-4bff-92a2-38160219bfb5-kube-api-access-szrlf\") pod \"maas-default-gateway-openshift-default-5bc884dd87-mrlcb\" (UID: \"73a30e41-ed57-4bff-92a2-38160219bfb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.695670 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.695609    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:52.973997 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:52.973976    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"]
Nov 01 21:52:52.975951 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:52:52.975926    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a30e41_ed57_4bff_92a2_38160219bfb5.slice/crio-90f195c0484853c26fa29b608ac1c2ced50be6527b9264913722a39a88275d36 WatchSource:0}: Error finding container 90f195c0484853c26fa29b608ac1c2ced50be6527b9264913722a39a88275d36: Status 404 returned error can't find the container with id 90f195c0484853c26fa29b608ac1c2ced50be6527b9264913722a39a88275d36
Nov 01 21:52:53.953132 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:53.953100    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" event={"ID":"73a30e41-ed57-4bff-92a2-38160219bfb5","Type":"ContainerStarted","Data":"90f195c0484853c26fa29b608ac1c2ced50be6527b9264913722a39a88275d36"}
Nov 01 21:52:57.224283 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:57.224244    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:52:57.224736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:57.224347    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:52:57.224736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:57.224401    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:52:57.983151 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:57.983107    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" event={"ID":"73a30e41-ed57-4bff-92a2-38160219bfb5","Type":"ContainerStarted","Data":"5448cb82a2495d967b51e6d02e9d912540db883a0e9dd01dc614881ab545b47c"}
Nov 01 21:52:58.008838 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:58.008787    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podStartSLOduration=1.763235141 podStartE2EDuration="6.008761098s" totalImagesPullingTime="4.245525957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:52:58.007330381 +0000 UTC m=+481.263510145" watchObservedRunningTime="2026-11-01 21:52:58.008761098 +0000 UTC m=+481.264940860"
Nov 01 21:52:58.696451 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:58.696414    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:52:58.697795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:58.697771    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:52:58.697856 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:58.697822    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:52:59.696274 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:59.696239    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:52:59.696472 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:52:59.696318    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:53:00.696220 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:00.696187    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:53:00.696413 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:00.696247    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:53:01.696893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:01.696856    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:53:01.697276 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:01.696919    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:53:02.696467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:02.696429    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:53:02.696620 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:02.696487    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:53:03.696404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:03.696367    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 01 21:53:03.696862 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:03.696430    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 01 21:53:04.817657 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:53:04.817621    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:53:05.463805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:05.463771    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:53:05.464219 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:05.464191    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:53:05.465153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:05.465132    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb"
Nov 01 21:53:07.383197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.383164    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 21:53:07.388041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.388012    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.402261 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.402237    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 21:53:07.414265 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.414238    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 21:53:07.418463 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.418441    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.432773 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.432742    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 21:53:07.562884 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.562853    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.562884 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.562886    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.562907    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.562926    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.562997    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldrv\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563151    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563186    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsp9\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563207    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563407 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563268    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563407 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563362    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563402    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563458    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563554 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563490    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563554 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563520    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563554 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563539    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.563715 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563658    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563715 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563695    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.563802 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.563767    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664134    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664168    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664201    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.664418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664250    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.664418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664287    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664372    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664404    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664695 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664656    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664706    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.664968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664902    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.664968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664930    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldrv\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664724    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664970    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.664967    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665032    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsp9\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665062    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665096    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665145    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665178    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665194    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665214    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665264    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665279    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665559    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.665709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665560    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.665709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.665646    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.666026 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.666004    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.666146 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.666126    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.666854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.666835    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.667188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.667168    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.667242 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.667227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.667474 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.667453    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.677788 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.677768    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldrv\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.677895 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.677768    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsp9\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.678010 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.677989    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-7bg9d\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.678230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.678214    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-5wxxh\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.704004 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.703983    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:07.732268 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.732243    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:07.984153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.984130    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 21:53:07.986140 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:07.986113    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b5e520_f8b0_41b6_91a9_7b030654b9e4.slice/crio-82a4a37194d45dfae56863b87012ce8c4dae39dd0438f6b795893a36733b32e1 WatchSource:0}: Error finding container 82a4a37194d45dfae56863b87012ce8c4dae39dd0438f6b795893a36733b32e1: Status 404 returned error can't find the container with id 82a4a37194d45dfae56863b87012ce8c4dae39dd0438f6b795893a36733b32e1
Nov 01 21:53:07.989975 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.989945    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:07.990040 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.990007    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:07.990040 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:07.990036    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:08.021272 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.021249    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 21:53:08.024375 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:08.024352    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2082417d_f041_4d19_84b5_1161d41cddd1.slice/crio-b04ee006d57452ea7d6c2ae9797887118ccafc3a9c9e85654eaa576d6599c35f WatchSource:0}: Error finding container b04ee006d57452ea7d6c2ae9797887118ccafc3a9c9e85654eaa576d6599c35f: Status 404 returned error can't find the container with id b04ee006d57452ea7d6c2ae9797887118ccafc3a9c9e85654eaa576d6599c35f
Nov 01 21:53:08.027599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.027569    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:08.027708 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.027645    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:08.027708 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.027701    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:08.036477 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.036413    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" event={"ID":"2082417d-f041-4d19-84b5-1161d41cddd1","Type":"ContainerStarted","Data":"b04ee006d57452ea7d6c2ae9797887118ccafc3a9c9e85654eaa576d6599c35f"}
Nov 01 21:53:08.038130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:08.038108    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" event={"ID":"59b5e520-f8b0-41b6-91a9-7b030654b9e4","Type":"ContainerStarted","Data":"82a4a37194d45dfae56863b87012ce8c4dae39dd0438f6b795893a36733b32e1"}
Nov 01 21:53:09.053322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.048472    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" event={"ID":"2082417d-f041-4d19-84b5-1161d41cddd1","Type":"ContainerStarted","Data":"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"}
Nov 01 21:53:09.072321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.069980    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" event={"ID":"59b5e520-f8b0-41b6-91a9-7b030654b9e4","Type":"ContainerStarted","Data":"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"}
Nov 01 21:53:09.096322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.091417    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podStartSLOduration=2.091395083 podStartE2EDuration="2.091395083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:09.07993937 +0000 UTC m=+492.336119134" watchObservedRunningTime="2026-11-01 21:53:09.091395083 +0000 UTC m=+492.347574848"
Nov 01 21:53:09.111488 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.111429    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podStartSLOduration=2.111412575 podStartE2EDuration="2.111412575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:09.111408504 +0000 UTC m=+492.367588266" watchObservedRunningTime="2026-11-01 21:53:09.111412575 +0000 UTC m=+492.367592338"
Nov 01 21:53:09.704209 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.704161    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:09.705823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.705795    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:09.705960 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.705841    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:09.733086 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.733058    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:09.734063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.734041    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:09.734138 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:09.734088    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:10.705029 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:10.704992    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:10.705452 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:10.705046    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:10.733377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:10.733349    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:10.733463 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:10.733397    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:11.705043 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:11.705003    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:11.705469 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:11.705076    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:11.733306 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:11.733266    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:11.733433 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:11.733327    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:12.704808 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:12.704764    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:12.705008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:12.704825    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:12.732738 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:12.732703    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:12.733176 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:12.732764    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:13.704680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:13.704630    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:13.704858 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:13.704695    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:13.733504 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:13.733471    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:13.733860 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:13.733515    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:14.704996 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:14.704951    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 01 21:53:14.705325 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:14.705004    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 01 21:53:14.732525 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:14.732494    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 01 21:53:14.732659 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:14.732538    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 01 21:53:16.378014 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.377975    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:16.378655 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.378566    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:16.379277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.379256    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 21:53:16.387116 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.387093    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:16.387359 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.387336    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:16.388130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:16.388115    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 21:53:34.813751 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:53:34.813715    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:53:47.424105 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.424071    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 21:53:47.434745 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.434719    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.445071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.444966    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 21:53:47.452397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.452375    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f3a6b-openshift-default-dockercfg-nlg9b\""
Nov 01 21:53:47.475801 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.475768    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.475927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.475827    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.475996 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.475970    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476045 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476030    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6g4h\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476115 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476095    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476156    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476201 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476184    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476233 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476200    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.476233 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.476223    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.508808 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.508777    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 21:53:47.512590 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.512563    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.528868 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.528840    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 21:53:47.552913 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.552878    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fb9597-openshift-default-dockercfg-lhqww\""
Nov 01 21:53:47.577482 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577447    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577513    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6g4h\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577545    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5869w\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.577618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577564    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.577618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577604    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577785    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577885 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577848    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577947 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577882    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.577947 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577908    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.577947 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577926    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578101 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577948    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578101 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.577977    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578101 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578009    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578101 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578094    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578249 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578161    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578249 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578194    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578249 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578214    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578249 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578216    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578365    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578404    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578458    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.578615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578501    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.578661 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.578632    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.580508 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.580489    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.580568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.580497    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.594542 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.594517    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6g4h\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.597472 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.597147    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.679623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679536    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.679623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679593    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.679623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679622    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.679897 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679673    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.679897 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679803    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.679897 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679858    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.679909    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680088    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680133    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5869w\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680346 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680168    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680346 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680222    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680616 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680594    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.680694 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680635    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.681018 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.680993    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.682277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.682259    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.682520 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.682504    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.697572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.697545    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.697856 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.697839    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5869w\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.710176 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.710156    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 21:53:47.715203 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.715187    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.727935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.727913    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 21:53:47.728527 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.728502    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b0b7a3-openshift-default-dockercfg-rxv7b\""
Nov 01 21:53:47.753216 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.753189    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:47.781724 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.781699    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.781828 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.781737    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.781914 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.781889    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782021 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782004    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782073 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782035    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782073 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782061    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzd46\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782172    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782227 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782209    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.782261 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.782236    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.830365 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.830326    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883611    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883728    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883774    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883808    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883858    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883947    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.883988    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.884031    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzd46\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.884789    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.885082    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.886847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.886586    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.888009 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.887987    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.888188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.888169    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.888593 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.888557    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.894553 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.894498    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 21:53:47.899924 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.898650    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.903556 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.903125    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.905909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.905887    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.906516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.906478    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzd46\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:47.912128 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.912101    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 21:53:47.912631 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.912156    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ff3e70-openshift-default-dockercfg-8tnvh\""
Nov 01 21:53:47.984990 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.984962    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985139 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985036    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985139 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985067    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985250 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985181    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985250 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985213    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2qm\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985387 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985293    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985387 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985368    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985475 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985406    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:47.985475 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:47.985460    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.030109 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.030082    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:48.036978 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.036955    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 21:53:48.043152 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:48.043117    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42001214_96ab_4dc5_a887_cc32182431b2.slice/crio-298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1 WatchSource:0}: Error finding container 298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1: Status 404 returned error can't find the container with id 298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1
Nov 01 21:53:48.045887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.045853    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.045992 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.045936    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.045992 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.045982    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087460    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087507    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2qm\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087569    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087600    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087633    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087772    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087793    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087848    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.087869    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.088833 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.088469    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.089554 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.089510    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.089698 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.089669    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.090211 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.089724    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.091175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.091157    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.093239 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.093214    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.103687 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.103652    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.106184 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.105715    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2qm\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.128490 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.128465    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 21:53:48.130546 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:48.130516    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e835814_8b13_4f31_b151_3981d24740b2.slice/crio-d82bd3bcfa4a386867e33872598580133507023cf7e0b0c1113c8ede27441d52 WatchSource:0}: Error finding container d82bd3bcfa4a386867e33872598580133507023cf7e0b0c1113c8ede27441d52: Status 404 returned error can't find the container with id d82bd3bcfa4a386867e33872598580133507023cf7e0b0c1113c8ede27441d52
Nov 01 21:53:48.135117 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.135075    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.135205 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.135172    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.135244 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.135219    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.219010 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.218980    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:48.278691 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.278657    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerStarted","Data":"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"}
Nov 01 21:53:48.278816 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.278726    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerStarted","Data":"d82bd3bcfa4a386867e33872598580133507023cf7e0b0c1113c8ede27441d52"}
Nov 01 21:53:48.282109 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.281708    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerStarted","Data":"426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2"}
Nov 01 21:53:48.282109 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.281762    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerStarted","Data":"298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1"}
Nov 01 21:53:48.310633 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.310575    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podStartSLOduration=1.310555514 podStartE2EDuration="1.310555514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:48.302228689 +0000 UTC m=+531.558408453" watchObservedRunningTime="2026-11-01 21:53:48.310555514 +0000 UTC m=+531.566735278"
Nov 01 21:53:48.328492 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.328363    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podStartSLOduration=1.328345473 podStartE2EDuration="1.328345473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:48.327885186 +0000 UTC m=+531.584064975" watchObservedRunningTime="2026-11-01 21:53:48.328345473 +0000 UTC m=+531.584525236"
Nov 01 21:53:48.335511 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.335485    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 21:53:48.337191 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:48.337138    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dba715_6fc8_428c_9f1a_1febdabc712e.slice/crio-6c075b8db23893027867546e588ebcdcf8400f436b147bd6831e3715e2fea83a WatchSource:0}: Error finding container 6c075b8db23893027867546e588ebcdcf8400f436b147bd6831e3715e2fea83a: Status 404 returned error can't find the container with id 6c075b8db23893027867546e588ebcdcf8400f436b147bd6831e3715e2fea83a
Nov 01 21:53:48.345424 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.345391    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.345524 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.345457    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.345524 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.345495    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.537064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.537039    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 21:53:48.539616 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:48.539582    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8681913_b53a_4ac8_8dd0_d5a239356972.slice/crio-b31a8ee53ebc9f4e80fe1a2059172de411b8b831427cb27be82ed29df8bc2a4c WatchSource:0}: Error finding container b31a8ee53ebc9f4e80fe1a2059172de411b8b831427cb27be82ed29df8bc2a4c: Status 404 returned error can't find the container with id b31a8ee53ebc9f4e80fe1a2059172de411b8b831427cb27be82ed29df8bc2a4c
Nov 01 21:53:48.542460 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.542431    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.542540 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.542489    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.542540 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.542518    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:53:48.753484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.753454    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:48.830585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:48.830491    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:49.288171 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.288124    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerStarted","Data":"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"}
Nov 01 21:53:49.288380 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.288234    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerStarted","Data":"6c075b8db23893027867546e588ebcdcf8400f436b147bd6831e3715e2fea83a"}
Nov 01 21:53:49.289893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.289856    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerStarted","Data":"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"}
Nov 01 21:53:49.290013 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.289914    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerStarted","Data":"b31a8ee53ebc9f4e80fe1a2059172de411b8b831427cb27be82ed29df8bc2a4c"}
Nov 01 21:53:49.313676 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.313627    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podStartSLOduration=2.313615431 podStartE2EDuration="2.313615431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:49.311458612 +0000 UTC m=+532.567638376" watchObservedRunningTime="2026-11-01 21:53:49.313615431 +0000 UTC m=+532.569795193"
Nov 01 21:53:49.335860 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.335815    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podStartSLOduration=2.335804573 podStartE2EDuration="2.335804573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:49.334203439 +0000 UTC m=+532.590383197" watchObservedRunningTime="2026-11-01 21:53:49.335804573 +0000 UTC m=+532.591984338"
Nov 01 21:53:49.759406 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.759377    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:49.836629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:49.836598    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:50.030576 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.030501    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:50.035977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.035953    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:50.219552 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.219527    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:50.224640 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.224619    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:50.296777 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296705    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:50.296777 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296762    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:50.296980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296790    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:50.296980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296815    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:50.296980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296875    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:53:50.296980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296904    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:53:50.296980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.296932    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:53:50.297230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:50.297071    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:53:53.360807 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.360774    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 21:53:53.364757 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.364736    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.368375 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.368354    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1f3a6b-serving-cert\""
Nov 01 21:53:53.374645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.374620    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 21:53:53.447213 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.447179    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pr8\" (UniqueName: \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.447419 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.447231    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.447419 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.447363    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.447494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.447443    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.548703 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.548665    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pr8\" (UniqueName: \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.548883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.548715    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.548883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.548757    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.548883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.548781    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.549455 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.549422    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.551592 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.551569    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.551714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.551653    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.561070 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.561044    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pr8\" (UniqueName: \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\") pod \"maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.680149 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.680061    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:53.962288 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.962265    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 21:53:53.964345 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:53.964292    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a8cdcb_ab23_41ec_a53f_c518a55319f5.slice/crio-ea3995e6bee4b300662c8fa0d5970ef831b272248fca5490227eec651ed6b57b WatchSource:0}: Error finding container ea3995e6bee4b300662c8fa0d5970ef831b272248fca5490227eec651ed6b57b: Status 404 returned error can't find the container with id ea3995e6bee4b300662c8fa0d5970ef831b272248fca5490227eec651ed6b57b
Nov 01 21:53:53.965713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:53.965696    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:53:54.331415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:54.331324    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" event={"ID":"c5a8cdcb-ab23-41ec-a53f-c518a55319f5","Type":"ContainerStarted","Data":"ea3995e6bee4b300662c8fa0d5970ef831b272248fca5490227eec651ed6b57b"}
Nov 01 21:53:55.338013 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.337974    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" event={"ID":"c5a8cdcb-ab23-41ec-a53f-c518a55319f5","Type":"ContainerStarted","Data":"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"}
Nov 01 21:53:55.338509 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.338031    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:53:55.358137 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.358062    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" podStartSLOduration=1.89165588 podStartE2EDuration="2.358047716s" totalImagesPullingTime="466.391836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:55.357788134 +0000 UTC m=+538.613967898" watchObservedRunningTime="2026-11-01 21:53:55.358047716 +0000 UTC m=+538.614227479"
Nov 01 21:53:55.783273 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.783242    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 21:53:55.787253 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.787230    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.794133 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.794109    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-fb9597-serving-cert\""
Nov 01 21:53:55.800145 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.800121    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 21:53:55.876013 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.875984    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.876160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.876031    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.876160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.876107    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.876243 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.876225    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhc9\" (UniqueName: \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.977808 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.977777    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.977967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.977870    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhc9\" (UniqueName: \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.977967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.977926    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.977967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.977964    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.978646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.978620    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.980310 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.980276    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.980429 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.980279    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:55.990007 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:55.989985    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhc9\" (UniqueName: \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\") pod \"maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:56.102008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:56.101927    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:56.388179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:56.388155    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 21:53:56.389730 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:56.389706    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b19f8a9_1528_4a10_8935_afb3a4ef2ee7.slice/crio-b2e74732e28fc631cbce1e072b0a0efd382bafea305d075701808fc721165a9d WatchSource:0}: Error finding container b2e74732e28fc631cbce1e072b0a0efd382bafea305d075701808fc721165a9d: Status 404 returned error can't find the container with id b2e74732e28fc631cbce1e072b0a0efd382bafea305d075701808fc721165a9d
Nov 01 21:53:57.365318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:57.364066    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" event={"ID":"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7","Type":"ContainerStarted","Data":"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"}
Nov 01 21:53:57.365318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:57.364136    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" event={"ID":"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7","Type":"ContainerStarted","Data":"b2e74732e28fc631cbce1e072b0a0efd382bafea305d075701808fc721165a9d"}
Nov 01 21:53:57.365318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:57.365114    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:53:57.386667 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:57.386620    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" podStartSLOduration=2.021637169 podStartE2EDuration="2.386607328s" totalImagesPullingTime="364.970159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:53:57.385806399 +0000 UTC m=+540.641986163" watchObservedRunningTime="2026-11-01 21:53:57.386607328 +0000 UTC m=+540.642787110"
Nov 01 21:53:59.266374 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.266334    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 21:53:59.274619 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.274596    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.278909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.278860    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0def36-serving-cert\""
Nov 01 21:53:59.280192 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.280168    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 21:53:59.315260 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.315224    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.315440 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.315338    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.315440 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.315372    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.315568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.315536    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbf5\" (UniqueName: \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.416693 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.416664    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbf5\" (UniqueName: \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.416854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.416752    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.416909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.416853    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.416909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.416885    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.417645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.417615    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.419098 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.419079    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.419380 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.419356    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.429617 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.429594    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbf5\" (UniqueName: \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\") pod \"maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.589638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.589553    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:53:59.874689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:53:59.874607    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 21:53:59.879550 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:53:59.879521    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565fc23b_8273_44b8_92c7_eb902f590022.slice/crio-425c64b9cf7d76c1c1942e955c100d2f5c52c17d7e7f296c04c4ff49f9fe8283 WatchSource:0}: Error finding container 425c64b9cf7d76c1c1942e955c100d2f5c52c17d7e7f296c04c4ff49f9fe8283: Status 404 returned error can't find the container with id 425c64b9cf7d76c1c1942e955c100d2f5c52c17d7e7f296c04c4ff49f9fe8283
Nov 01 21:54:00.073656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.063411    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 21:54:00.094435 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.094408    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 21:54:00.094604 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.094541    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.100334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.099469    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ff3e70-serving-cert\""
Nov 01 21:54:00.124012 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.123986    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.124127 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.124049    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.124127 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.124105    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.124196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.124138    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwmw\" (UniqueName: \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.225483 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.225452    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.225688 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.225530    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.225688 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.225557    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.225688 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.225578    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwmw\" (UniqueName: \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.226226 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.226205    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.227956 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.227934    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.228037 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.227965    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.239907 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.239879    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwmw\" (UniqueName: \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\") pod \"maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.396077 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.396025    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" event={"ID":"565fc23b-8273-44b8-92c7-eb902f590022","Type":"ContainerStarted","Data":"425c64b9cf7d76c1c1942e955c100d2f5c52c17d7e7f296c04c4ff49f9fe8283"}
Nov 01 21:54:00.411136 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.410617    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:00.723287 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:00.723249    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 21:54:01.373488 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.373460    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 21:54:01.405571 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.405526    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" event={"ID":"0e36822e-f396-4ebe-a7b9-0b3a967446c6","Type":"ContainerStarted","Data":"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"}
Nov 01 21:54:01.406076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.405607    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" event={"ID":"0e36822e-f396-4ebe-a7b9-0b3a967446c6","Type":"ContainerStarted","Data":"ac340f8d33f1d7e83208f03b21b2499511d6c43b166fec7b14667cd612c0a340"}
Nov 01 21:54:01.406076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.405671    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:01.409397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.408783    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" event={"ID":"565fc23b-8273-44b8-92c7-eb902f590022","Type":"ContainerStarted","Data":"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"}
Nov 01 21:54:01.409397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.408935    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:54:01.479011 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:01.474813    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" podStartSLOduration=0.955459373 podStartE2EDuration="1.47478909s" totalImagesPullingTime="519.329717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:01.428395673 +0000 UTC m=+544.684575436" watchObservedRunningTime="2026-11-01 21:54:01.47478909 +0000 UTC m=+544.730968856"
Nov 01 21:54:04.383009 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:04.382981    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 21:54:04.403950 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:04.403897    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" podStartSLOduration=4.987344988 podStartE2EDuration="5.403876945s" totalImagesPullingTime="416.531957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:01.47548304 +0000 UTC m=+544.731662803" watchObservedRunningTime="2026-11-01 21:54:04.403876945 +0000 UTC m=+547.660056709"
Nov 01 21:54:04.837639 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:54:04.837590    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:54:06.338510 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:06.338454    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:54:06.339019 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:06.338527    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:54:07.420804 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:07.420770    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 21:54:07.421648 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:07.421612    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 21:54:09.128880 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:54:09.128831    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@0-13737-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42001214_96ab_4dc5_a887_cc32182431b2.slice/crio-426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 21:54:09.533094 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.533056    5845 generic.go:356] "Generic (PLEG): container finished" podID="42001214-96ab-4dc5-a887-cc32182431b2" containerID="426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2" exitCode=0
Nov 01 21:54:09.533281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.533108    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerDied","Data":"426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2"}
Nov 01 21:54:09.533661 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.533644    5845 scope.go:122] "RemoveContainer" containerID="426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2"
Nov 01 21:54:09.535338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.535288    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:09.535421 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.535400    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:09.535464 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:09.535453    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:10.540085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:10.540040    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerStarted","Data":"eab42bcbabf7447de6f73c88712667268eb4ace915f644aedc2c911f5bfec3bc"}
Nov 01 21:54:10.761323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:10.760801    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:54:10.761323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:10.760961    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:10.761323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:10.761021    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:11.768083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:11.767627    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:11.768083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:11.767695    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:12.754212 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:12.753690    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:12.754212 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:12.753754    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:13.758323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:13.754852    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:13.758323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:13.754925    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:14.755782 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:14.755746    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:14.756081 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:14.756037    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:15.757791 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:15.756207    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 01 21:54:15.757791 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:15.756269    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 01 21:54:16.620997 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.620964    5845 generic.go:356] "Generic (PLEG): container finished" podID="2e835814-8b13-4f31-b151-3981d24740b2" containerID="f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f" exitCode=0
Nov 01 21:54:16.643690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.621041    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerDied","Data":"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"}
Nov 01 21:54:16.643690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.621469    5845 scope.go:122] "RemoveContainer" containerID="f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"
Nov 01 21:54:16.643690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.622802    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:16.643690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.622863    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:16.643690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.622895    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:16.758672 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.758648    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:54:16.759130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.759113    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:54:16.759886 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:16.759865    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 21:54:17.628132 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.628102    5845 generic.go:356] "Generic (PLEG): container finished" podID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerID="e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d" exitCode=0
Nov 01 21:54:17.628414 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.628178    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerDied","Data":"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"}
Nov 01 21:54:17.628715 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.628697    5845 scope.go:122] "RemoveContainer" containerID="e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"
Nov 01 21:54:17.630366 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.630336    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.630473 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.630443    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.630516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.630481    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerStarted","Data":"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"}
Nov 01 21:54:17.630516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.630494    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.632232 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.632212    5845 generic.go:356] "Generic (PLEG): container finished" podID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerID="761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317" exitCode=0
Nov 01 21:54:17.632342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.632320    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerDied","Data":"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"}
Nov 01 21:54:17.632817 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.632800    5845 scope.go:122] "RemoveContainer" containerID="761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"
Nov 01 21:54:17.633973 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.633948    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.634049 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.634009    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.634049 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.634040    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:54:17.835340 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.831197    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:54:17.835340 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.834438    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:17.835340 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:17.834489    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:18.639980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:18.639943    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerStarted","Data":"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"}
Nov 01 21:54:18.641981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:18.641953    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerStarted","Data":"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"}
Nov 01 21:54:18.831314 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:18.831264    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:18.831487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:18.831334    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:19.030854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.030821    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:54:19.031498 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.031472    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 21:54:19.031608 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.031519    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 21:54:19.221460 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.221423    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:54:19.221820 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.221794    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 21:54:19.221981 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.221956    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 21:54:19.887334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.884865    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:19.887334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:19.884921    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:20.031953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.031921    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 21:54:20.032429 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.032400    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 21:54:20.033116 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.033090    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 21:54:20.038214 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.038190    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.073347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.066867    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 21:54:20.073347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.067094    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"kube-root-ca.crt\""
Nov 01 21:54:20.073347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.067243    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"openshift-service-ca.crt\""
Nov 01 21:54:20.096856 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.096808    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.097149 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.097120    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.097385 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.097363    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthxb\" (UniqueName: \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.097565 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.097546    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.101437 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.101408    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.101615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.101596    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.131327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.102622    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"default-dockercfg-rr98w\""
Nov 01 21:54:20.131327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.123520    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"simulator-w5-b0b7a3-kserve-self-signed-certs\""
Nov 01 21:54:20.240599 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.240565    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.241347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.241323    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.242086 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.242065    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthxb\" (UniqueName: \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.242684 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.242665    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.243181 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.243162    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.243721 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.243704    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.248439 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.243050    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.248589 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.241223    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.248748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.244110    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.248841 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.244663    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.248931 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.241944    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 21:54:20.249060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.249037    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 21:54:20.284319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.272198    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.302320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.298376    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:54:20.302320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.298459    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:54:20.302320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.298486    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:54:20.302320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.298379    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthxb\" (UniqueName: \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\") pod \"simulator-w5-b0b7a3-kserve-744646869f-7shvs\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") " pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.436482 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.436454    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:20.831068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.830664    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:20.831068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.830721    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:20.936214 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:20.936190    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 21:54:21.032392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.032364    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 21:54:21.032849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.032818    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 21:54:21.232323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.229821    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 21:54:21.232323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.229886    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 21:54:21.740637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.740534    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" event={"ID":"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6","Type":"ContainerStarted","Data":"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"}
Nov 01 21:54:21.740637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.740593    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" event={"ID":"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6","Type":"ContainerStarted","Data":"7d2fbd9cc0669c61d1b0a7e9c464ab4cc2c3f5f41c14b9761dd75d17501d1077"}
Nov 01 21:54:21.808038 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.807969    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podStartSLOduration=1.807946563 podStartE2EDuration="1.807946563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:21.792221881 +0000 UTC m=+565.048401646" watchObservedRunningTime="2026-11-01 21:54:21.807946563 +0000 UTC m=+565.064126326"
Nov 01 21:54:21.855406 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.832643    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:21.855719 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:21.855687    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:22.032696 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.031733    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 21:54:22.032696 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.031792    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 21:54:22.222269 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.221768    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 21:54:22.222269 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.221887    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 21:54:22.437290 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.437185    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:22.830634 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.830591    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 21:54:22.830840 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:22.830640    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 21:54:23.031247 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.031210    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 21:54:23.031792 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.031757    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 21:54:23.223482 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.223450    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 21:54:23.224070 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.224035    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 21:54:23.835880 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.835849    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:54:23.836900 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:23.836876    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 21:54:24.164087 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:24.164005    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:54:24.165866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:24.165841    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 21:54:24.225148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:24.225120    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:54:24.226396 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:24.226367    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 21:54:25.805465 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.805428    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 21:54:25.808377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.808353    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.812479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.812456    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"kube-root-ca.crt\""
Nov 01 21:54:25.819734 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.819704    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"simulator-w1-53b784-kserve-self-signed-certs\""
Nov 01 21:54:25.829234 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.829205    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"default-dockercfg-bpmhg\""
Nov 01 21:54:25.840864 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.840839    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 21:54:25.917106 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.917075    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"openshift-service-ca.crt\""
Nov 01 21:54:25.955717 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.955683    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.955877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.955742    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.955877 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.955828    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.956003 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.955977    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.956048 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.956018    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484dt\" (UniqueName: \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:25.956083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:25.956057    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057171    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057367    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057428    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484dt\" (UniqueName: \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057579    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057658    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.057696    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.058324    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.058555    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.059076 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.058662    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.062642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.062612    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.063881 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.063852    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.070100 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.070075    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-484dt\" (UniqueName: \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\") pod \"simulator-w1-53b784-kserve-76d55f9477-pl9k7\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") " pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.145416 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.145384    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:26.443126 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.442903    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 21:54:26.814081 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.813984    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" event={"ID":"62b0fab0-718a-497e-881d-70a837997119","Type":"ContainerStarted","Data":"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"}
Nov 01 21:54:26.814081 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.814041    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" event={"ID":"62b0fab0-718a-497e-881d-70a837997119","Type":"ContainerStarted","Data":"0415e61ef19e47dc09aad62f74a7e23809f3a926ead2c894280336b0e6c9c590"}
Nov 01 21:54:26.835188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:26.835138    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podStartSLOduration=1.835123323 podStartE2EDuration="1.835123323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:26.833501092 +0000 UTC m=+570.089680853" watchObservedRunningTime="2026-11-01 21:54:26.835123323 +0000 UTC m=+570.091303086"
Nov 01 21:54:27.145547 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:27.145461    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:28.378340 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.378276    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 21:54:28.381947 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.381929    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.396686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.392192    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"kube-root-ca.crt\""
Nov 01 21:54:28.396686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.392273    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"simulator-w2-f033b1-kserve-self-signed-certs\""
Nov 01 21:54:28.396686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.392379    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"default-dockercfg-pc87z\""
Nov 01 21:54:28.396686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.392962    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f033b1\"/\"openshift-service-ca.crt\""
Nov 01 21:54:28.396686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.394206    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 21:54:28.515441 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515399    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.515605 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515448    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.515647 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515597    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.515647 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515638    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf2ks\" (UniqueName: \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.515767 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515748    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.515805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.515794    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.616817 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.616775    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.616967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.616830    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf2ks\" (UniqueName: \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.616967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.616899    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.616967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.616930    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.617108 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.617008    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.617108 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.617039    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.617262 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.617239    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.617446 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.617420    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.617586 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.617453    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.619387 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.619358    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.619682 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.619664    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.632127 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.632078    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf2ks\" (UniqueName: \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\") pod \"simulator-w2-f033b1-kserve-557d698566-9kk5w\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") " pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.697829 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.697806    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:28.974385 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:28.974342    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 21:54:28.976689 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:54:28.976658    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad5419e4_dceb_4777_b863_cddeb99eb425.slice/crio-7860a84fdc4013bdcbb489f8a9269005f166ad1df44247eda85d9f65c64a3027 WatchSource:0}: Error finding container 7860a84fdc4013bdcbb489f8a9269005f166ad1df44247eda85d9f65c64a3027: Status 404 returned error can't find the container with id 7860a84fdc4013bdcbb489f8a9269005f166ad1df44247eda85d9f65c64a3027
Nov 01 21:54:29.838831 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:29.838790    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" event={"ID":"ad5419e4-dceb-4777-b863-cddeb99eb425","Type":"ContainerStarted","Data":"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"}
Nov 01 21:54:29.839225 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:29.838847    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" event={"ID":"ad5419e4-dceb-4777-b863-cddeb99eb425","Type":"ContainerStarted","Data":"7860a84fdc4013bdcbb489f8a9269005f166ad1df44247eda85d9f65c64a3027"}
Nov 01 21:54:29.859285 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:29.859216    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podStartSLOduration=1.8592025680000002 podStartE2EDuration="1.859202568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:29.858065792 +0000 UTC m=+573.114245556" watchObservedRunningTime="2026-11-01 21:54:29.859202568 +0000 UTC m=+573.115382355"
Nov 01 21:54:30.437274 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:30.437237    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:30.449864 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:30.449841    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:30.698453 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:30.698376    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:30.860914 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:30.860883    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:54:34.815719 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:54:34.815675    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:54:36.147258 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:36.147182    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:36.162083 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:36.162049    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:36.895719 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:36.895688    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:54:38.698935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:38.698904    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:38.711643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:38.711623    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:38.906027 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:38.906000    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:54:52.533537 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.533498    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 21:54:52.538480 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.538451    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.555943 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.555901    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"openshift-service-ca.crt\""
Nov 01 21:54:52.556229 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.556209    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"kube-root-ca.crt\""
Nov 01 21:54:52.571792 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.571768    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 21:54:52.574961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.574941    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"default-dockercfg-77f4g\""
Nov 01 21:54:52.575071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.575019    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"premium-w6-fb9597-kserve-self-signed-certs\""
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.607716    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.607800    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54clc\" (UniqueName: \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.607870    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.607904    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.607987    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.610497 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.608040    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.711975 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.711241    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.712368 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.712341    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.713153 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.713018    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.713486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.713467    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.714011 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.713899    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.714543 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.714510    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54clc\" (UniqueName: \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.714706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.714691    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.715141 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.715123    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.715712 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.715514    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.717392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.717284    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.718514 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.718489    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.727179 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.727155    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54clc\" (UniqueName: \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\") pod \"premium-w6-fb9597-kserve-565984d6-lhnrw\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") " pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:52.883501 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:52.883412    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:53.175058 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:53.174978    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 21:54:53.181175 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:54:53.181140    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fdc7f29_1c6d_4c2d_a758_9f58b124c5bf.slice/crio-d9c0a63e61cb5b5e7166ce2e246ff937a28a76c5c926880654bc5fc840aa4ad0 WatchSource:0}: Error finding container d9c0a63e61cb5b5e7166ce2e246ff937a28a76c5c926880654bc5fc840aa4ad0: Status 404 returned error can't find the container with id d9c0a63e61cb5b5e7166ce2e246ff937a28a76c5c926880654bc5fc840aa4ad0
Nov 01 21:54:54.006491 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:54.006454    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" event={"ID":"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf","Type":"ContainerStarted","Data":"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"}
Nov 01 21:54:54.006864 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:54.006504    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" event={"ID":"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf","Type":"ContainerStarted","Data":"d9c0a63e61cb5b5e7166ce2e246ff937a28a76c5c926880654bc5fc840aa4ad0"}
Nov 01 21:54:54.030736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:54.030683    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podStartSLOduration=2.030669877 podStartE2EDuration="2.030669877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:54:54.028025093 +0000 UTC m=+597.284204857" watchObservedRunningTime="2026-11-01 21:54:54.030669877 +0000 UTC m=+597.286849640"
Nov 01 21:54:54.883878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:54.883850    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:54:56.928347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:56.928319    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:54:56.929338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:54:56.929312    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:55:02.884358 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:55:02.884320    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:55:02.897202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:55:02.897176    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:55:03.078977 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:55:03.078948    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:55:04.815528 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:55:04.815487    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:55:34.815166 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:55:34.815127    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:56:04.814715 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:56:04.814675    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:56:11.012063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.012023    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 21:56:11.015939 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.015916    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.021542 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.021517    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fb9597\"/\"distinct-w6-fb9597-kserve-self-signed-certs\""
Nov 01 21:56:11.024253 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.024230    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 21:56:11.141404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141372    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.141593 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141505    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.141593 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141536    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.141593 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141566    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgr8k\" (UniqueName: \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.141742 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141633    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.141797 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.141772    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.242712 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.242882 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242723    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.242882 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242746    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgr8k\" (UniqueName: \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.242882 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242787    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.242882 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242833    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.243064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.242916    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.243246 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.243219    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.243397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.243218    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.243397 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.243253    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.244887 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.244864    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.245347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.245289    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.255400 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.255378    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgr8k\" (UniqueName: \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\") pod \"distinct-w6-fb9597-kserve-774b85cb69-mkcfk\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") " pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.329763 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.329703    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:11.606762 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:11.606725    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 21:56:11.610092 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:11.610054    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb321c879_897d_4097_b678_9b37339df479.slice/crio-4bf3452700875ff9bd45e0b4ccfbcbacaef3a0a8ef94bc6a86e6da156d958497 WatchSource:0}: Error finding container 4bf3452700875ff9bd45e0b4ccfbcbacaef3a0a8ef94bc6a86e6da156d958497: Status 404 returned error can't find the container with id 4bf3452700875ff9bd45e0b4ccfbcbacaef3a0a8ef94bc6a86e6da156d958497
Nov 01 21:56:12.523044 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:12.523006    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" event={"ID":"b321c879-897d-4097-b678-9b37339df479","Type":"ContainerStarted","Data":"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"}
Nov 01 21:56:12.523425 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:12.523056    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" event={"ID":"b321c879-897d-4097-b678-9b37339df479","Type":"ContainerStarted","Data":"4bf3452700875ff9bd45e0b4ccfbcbacaef3a0a8ef94bc6a86e6da156d958497"}
Nov 01 21:56:12.547771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:12.547719    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podStartSLOduration=2.547685714 podStartE2EDuration="2.547685714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:12.544807811 +0000 UTC m=+675.800987571" watchObservedRunningTime="2026-11-01 21:56:12.547685714 +0000 UTC m=+675.803865474"
Nov 01 21:56:13.329974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:13.329930    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:17.616312 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.616265    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 21:56:17.620345 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.620322    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.625900 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.625876    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0def36-openshift-default-dockercfg-wfkj8\""
Nov 01 21:56:17.632386 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.632367    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 21:56:17.709723 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709689    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.709883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709737    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.709883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709845    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.709883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709870    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.710007 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709891    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.710007 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.709973    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6vqc\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.710096 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.710025    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.710096 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.710042    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.710197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.710165    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811586 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811555    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811592    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811621    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811656    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6vqc\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811728    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.811771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811750    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812045 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811818    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812045 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.811898    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812045 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812029    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812106    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812121    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812267 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812185    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812402 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812378    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.812633 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.812607    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.814217 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.814195    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.814462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.814440    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.823787 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.823742    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.823885 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.823822    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6vqc\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:17.937056 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:17.936973    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:18.044212 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.043836    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 21:56:18.051678 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.051632    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.063745 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.063016    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-53b784-openshift-default-dockercfg-m5k4p\""
Nov 01 21:56:18.083808 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.082729    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 21:56:18.088336 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.087713    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 21:56:18.088336 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.087826    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.090136 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.090110    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 21:56:18.115559 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.115530    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm299\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.116010 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.115890    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.116927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.116898    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117108 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117096    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.117242 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117231    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.117419 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117393    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.117544 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117532    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117683 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117664    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117725 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117700    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.117758 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117726    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117756    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.117825 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117784    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117825 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117815    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.117885 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.117849    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.118041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.118020    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.118107 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.118071    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.118194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.118117    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.118194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.118155    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219095 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219062    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219103    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219120    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219157    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219420 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219255    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219420 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219319    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219420 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219377    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm299\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219443    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219469    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219528    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219569    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219569    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219608    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219631    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219778    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219815    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219847    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219872    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219902    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.219953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.219940    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.220482 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.220461    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.220630 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.220610    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.220706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.220632    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.220706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.220672    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.220794 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.220711    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.221051 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.221010    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.221205 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.221153    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.221267 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.221253    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.223204 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.223183    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.223741 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.223722    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.223879 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.223819    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.223946 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.223891    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.231321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.231277    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.231392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.231383    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm299\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.234277 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.234252    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.235361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.235342    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.248827 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.248197    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 21:56:18.251454 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:18.251432    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42810644_d18a_4428_a354_4801bae499d3.slice/crio-06c9351e9febaac4c6ede58fe31a71300ea3f43390dd652512e594fe54899d43 WatchSource:0}: Error finding container 06c9351e9febaac4c6ede58fe31a71300ea3f43390dd652512e594fe54899d43: Status 404 returned error can't find the container with id 06c9351e9febaac4c6ede58fe31a71300ea3f43390dd652512e594fe54899d43
Nov 01 21:56:18.253986 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.253950    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.254063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.254030    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.254100 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.254073    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.400019 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.399988    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:18.409593 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.409559    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:18.557047 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.557013    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" event={"ID":"42810644-d18a-4428-a354-4801bae499d3","Type":"ContainerStarted","Data":"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"}
Nov 01 21:56:18.557248 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.557077    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" event={"ID":"42810644-d18a-4428-a354-4801bae499d3","Type":"ContainerStarted","Data":"06c9351e9febaac4c6ede58fe31a71300ea3f43390dd652512e594fe54899d43"}
Nov 01 21:56:18.579687 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.579619    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podStartSLOduration=1.579600911 podStartE2EDuration="1.579600911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:18.578960242 +0000 UTC m=+681.835140489" watchObservedRunningTime="2026-11-01 21:56:18.579600911 +0000 UTC m=+681.835780676"
Nov 01 21:56:18.689235 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.689202    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 21:56:18.691261 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:18.691234    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb2beeb_625b_43f0_92d9_f7a55c3c1244.slice/crio-200d1c91b20db4edf951a08d7d63a95d051a18b7fc347eca23c0bec51e40c201 WatchSource:0}: Error finding container 200d1c91b20db4edf951a08d7d63a95d051a18b7fc347eca23c0bec51e40c201: Status 404 returned error can't find the container with id 200d1c91b20db4edf951a08d7d63a95d051a18b7fc347eca23c0bec51e40c201
Nov 01 21:56:18.694465 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.694432    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.694580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.694498    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.694580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.694527    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.695910 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.695884    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 21:56:18.698818 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:18.698789    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbd6256_a14f_4146_9993_6643d96ddee8.slice/crio-ab2bfe09507c171bae36b47b40acb707ca1ba99605b4a83e76db46f8779d34f1 WatchSource:0}: Error finding container ab2bfe09507c171bae36b47b40acb707ca1ba99605b4a83e76db46f8779d34f1: Status 404 returned error can't find the container with id ab2bfe09507c171bae36b47b40acb707ca1ba99605b4a83e76db46f8779d34f1
Nov 01 21:56:18.703585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.703560    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.703677 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.703616    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.703677 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.703642    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:18.937878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:18.937820    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:19.577319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.576596    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" event={"ID":"9fbd6256-a14f-4146-9993-6643d96ddee8","Type":"ContainerStarted","Data":"192beb8743743bb255e05521ff5a2ba51950f2a43d8940fd4526a969042ace28"}
Nov 01 21:56:19.577319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.576666    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" event={"ID":"9fbd6256-a14f-4146-9993-6643d96ddee8","Type":"ContainerStarted","Data":"ab2bfe09507c171bae36b47b40acb707ca1ba99605b4a83e76db46f8779d34f1"}
Nov 01 21:56:19.579917 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.579885    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" event={"ID":"9bb2beeb-625b-43f0-92d9-f7a55c3c1244","Type":"ContainerStarted","Data":"4b1b31a77dfb3296f07f0d419d7410f8dab671d24ef9fbb935df23dd48c84ad5"}
Nov 01 21:56:19.580148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.580128    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" event={"ID":"9bb2beeb-625b-43f0-92d9-f7a55c3c1244","Type":"ContainerStarted","Data":"200d1c91b20db4edf951a08d7d63a95d051a18b7fc347eca23c0bec51e40c201"}
Nov 01 21:56:19.602769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.602703    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podStartSLOduration=1.602683004 podStartE2EDuration="1.602683004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:19.602617964 +0000 UTC m=+682.858797739" watchObservedRunningTime="2026-11-01 21:56:19.602683004 +0000 UTC m=+682.858862768"
Nov 01 21:56:19.630942 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.630870    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podStartSLOduration=1.630848822 podStartE2EDuration="1.630848822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:19.62886508 +0000 UTC m=+682.885044854" watchObservedRunningTime="2026-11-01 21:56:19.630848822 +0000 UTC m=+682.887028585"
Nov 01 21:56:19.968080 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.967617    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:19.968080 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:19.967724    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:20.133161 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.129376    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 21:56:20.137486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.134892    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.140777    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.140862    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.140914    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.140945    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.141042    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zghx\" (UniqueName: \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.141713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.141084    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.166766    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 21:56:20.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.167209    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"openshift-service-ca.crt\""
Nov 01 21:56:20.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.167474    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"distinct-w4-0def36-kserve-self-signed-certs\""
Nov 01 21:56:20.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.168174    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"default-dockercfg-kwc4v\""
Nov 01 21:56:20.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.168281    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"kube-root-ca.crt\""
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245086    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245174    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245219    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245252    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245424    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zghx\" (UniqueName: \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.245474    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.248384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.246081    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.365194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.365115    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.401334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.396030    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.403415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.403384    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:20.404019 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.403999    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.435388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.407996    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zghx\" (UniqueName: \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.435388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.412816    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\") pod \"distinct-w4-0def36-kserve-5b8f9dcd99-x9l58\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.435388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.425642    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:20.468322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.442882    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:20.469580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.442915    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:20.507364 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.487167    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:20.507364 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.506410    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:20.529321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.469546    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:20.975006 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.971672    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:20.975006 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:20.971751    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:21.400349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.375887    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:21.461322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.402104    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:21.461322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.402177    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:21.461322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.412381    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:21.461322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.412451    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:21.461322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.455636    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 21:56:21.738121 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.737352    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" event={"ID":"3a4cc922-e1aa-4edf-bc11-26809fd8cd10","Type":"ContainerStarted","Data":"c6aab56c7f0cb593379042ec24e5d149969cc08555157248cff7438a5e369498"}
Nov 01 21:56:21.900953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.867810    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:21.900953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.879421    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:56:21.944958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.934735    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.021316 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.993054    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:22.021316 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:21.993113    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:22.074564 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.073397    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.370772 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.370401    5845 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.409322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.400678    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:22.409322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.400738    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:22.414319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.410332    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:22.414319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.410381    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:22.958379 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.958325    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:56:22.961479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.961443    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.969347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.965410    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.985201 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.969819    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.986206 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.969843    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:22.990316 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.969934    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:22.990524 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:22.990478    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:23.090429 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.090381    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:23.101676 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.090949    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:56:23.101956 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.101936    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:56:23.102106 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.102093    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:56:23.102237 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.102224    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:56:23.486768 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.484425    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:23.486768 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.484489    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:23.487079 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.487054    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:23.487215 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.487193    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:23.945378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.941416    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:23.945378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.941475    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:23.962320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.961390    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 01 21:56:23.962320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.961517    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:56:23.962320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.961890    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:56:24.004321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.988373    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:24.004321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.988769    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:56:24.004321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:23.989807    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:24.104434 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.104386    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:24.501401 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.417761    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:24.501401 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.417812    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:24.501401 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.418377    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:24.501401 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.418418    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:24.849389 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.800458    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:56:24.984321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.948999    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:24.984321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.949064    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:24.990461 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.990405    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:25.003742 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.985278    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:25.007322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.990690    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:25.110381 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:24.990719    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:25.418689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.411489    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:25.419759 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.419715    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:25.435554 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.435522    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:25.435813 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.435783    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:25.521325 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.518809    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" event={"ID":"3a4cc922-e1aa-4edf-bc11-26809fd8cd10","Type":"ContainerStarted","Data":"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"}
Nov 01 21:56:25.554367 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.551475    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podStartSLOduration=5.55145279 podStartE2EDuration="5.55145279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:25.549706266 +0000 UTC m=+688.805886030" watchObservedRunningTime="2026-11-01 21:56:25.55145279 +0000 UTC m=+688.807632553"
Nov 01 21:56:25.801424 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.801375    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 21:56:25.953342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.938666    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:25.953342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:25.938723    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:26.004321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.003483    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:56:26.005323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.004631    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:56:26.102383 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.099405    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:26.128321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.125386    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:26.416392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.416276    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:26.416645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.416614    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:26.417473 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.417436    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:26.417638 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.417620    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:26.491130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.491095    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:26.606319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.605601    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:56:26.876318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.841254    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:26.876318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.841582    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:56:26.940548 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.940503    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 21:56:26.940837 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.940801    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 21:56:26.941766 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:26.941701    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:27.036371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.022049    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:27.110323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.106378    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:27.126606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.126467    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:27.141239 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.139620    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:56:27.228324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.225397    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:27.323427 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.323397    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:56:27.336257 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.336228    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:56:27.342662 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.342634    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:56:27.408380 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.407861    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 01 21:56:27.408380 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.407916    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 01 21:56:27.417467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.417437    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:27.417709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:27.417681    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:28.410533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.410499    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 01 21:56:28.411626 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.411114    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 01 21:56:28.780320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.779287    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:28.781338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.780799    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:28.782043 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.782020    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 21:56:28.981100 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.981064    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:56:28.982722 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:28.982683    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:56:29.013899 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.013868    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:29.014263 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.014236    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:29.015396 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.015375    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 21:56:29.935514 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.935471    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:29.937009 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.936981    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:29.937793 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:29.937761    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 21:56:30.488306 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:30.488257    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:30.500721 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:30.500692    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:30.646527 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:30.646496    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:56:31.341124 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:31.341092    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:31.352359 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:31.352334    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:56:34.818098 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:56:34.818058    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:56:35.968240 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:35.968204    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 21:56:35.972880 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:35.972858    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:35.979488 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:35.979464    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-53b784\"/\"distinct-w1-53b784-kserve-self-signed-certs\""
Nov 01 21:56:35.981824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:35.981803    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 21:56:36.069476 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069447    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.069637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069488    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pvx\" (UniqueName: \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.069637 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069519    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.069704 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069642    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.069704 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069683    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.069767 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.069742    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.170942 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.170899    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.171122 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.170977    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.171122 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.171094    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.171270 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.171140    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pvx\" (UniqueName: \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.171270 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.171190    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.171396 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.171277    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.172099 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.172070    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.172700 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.172349    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.173635 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.173611    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.174915 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.174890    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.175272 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.175242    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.185154 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.185126    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pvx\" (UniqueName: \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\") pod \"distinct-w1-53b784-kserve-5d545c4ff9-qb8zz\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") " pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.287122 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.287046    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:36.568748 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.568722    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 21:56:36.570558 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:36.570530    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb43ab1_4a4a_4d10_8245_7c0d275d2813.slice/crio-e660a1ddc562772668d1fab9fcea64f44cdd06f0087d5ebb308e1aa1fae4bcde WatchSource:0}: Error finding container e660a1ddc562772668d1fab9fcea64f44cdd06f0087d5ebb308e1aa1fae4bcde: Status 404 returned error can't find the container with id e660a1ddc562772668d1fab9fcea64f44cdd06f0087d5ebb308e1aa1fae4bcde
Nov 01 21:56:36.670731 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.670698    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" event={"ID":"4cb43ab1-4a4a-4d10-8245-7c0d275d2813","Type":"ContainerStarted","Data":"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"}
Nov 01 21:56:36.670866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.670739    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" event={"ID":"4cb43ab1-4a4a-4d10-8245-7c0d275d2813","Type":"ContainerStarted","Data":"e660a1ddc562772668d1fab9fcea64f44cdd06f0087d5ebb308e1aa1fae4bcde"}
Nov 01 21:56:36.691894 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:36.691838    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podStartSLOduration=1.691825604 podStartE2EDuration="1.691825604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:36.68942688 +0000 UTC m=+699.945606643" watchObservedRunningTime="2026-11-01 21:56:36.691825604 +0000 UTC m=+699.948005367"
Nov 01 21:56:37.287606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:37.287571    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:46.287252 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:46.287221    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:46.299404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:46.299385    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:46.770822 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:46.770793    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:56:47.886803 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.886768    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 21:56:47.891008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.890986    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.895841 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.895816    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f033b1-openshift-default-dockercfg-svg6v\""
Nov 01 21:56:47.903352 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.903327    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 21:56:47.996863 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.996826    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bjh\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.996956    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.996986    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997038    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997237 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997201    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997371 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997351    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997439 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997381    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997439 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997411    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:47.997587 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:47.997567    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099449 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099415    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099483    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099503    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099519    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099606 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099536    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099632    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bjh\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099692    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099734    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.099849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.099786    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.100055 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.100030    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.100223 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.100202    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.100313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.100252    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.100313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.100271    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.100437 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.100288    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.101683 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.101663    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.102038 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.102021    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.111355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.111293    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.111450 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.111427    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bjh\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\") pod \"e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") " pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.207074 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.207000    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:48.491087 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.491058    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 21:56:48.492179 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:48.492152    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c93c32_5378_4a0d_97e9_dda679b6aad0.slice/crio-6f1fe807ada5e048043d843228d95feaa9f568f0fd4d0c4dd95dde8b083d06e5 WatchSource:0}: Error finding container 6f1fe807ada5e048043d843228d95feaa9f568f0fd4d0c4dd95dde8b083d06e5: Status 404 returned error can't find the container with id 6f1fe807ada5e048043d843228d95feaa9f568f0fd4d0c4dd95dde8b083d06e5
Nov 01 21:56:48.494556 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.494519    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:48.494668 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.494588    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:48.494668 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.494618    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:56:48.787726 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.787607    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" event={"ID":"b1c93c32-5378-4a0d-97e9-dda679b6aad0","Type":"ContainerStarted","Data":"6c45d3b91641321fa996b78023ed33c471107aa84991b4aabbb6be5da942752e"}
Nov 01 21:56:48.787726 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.787669    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" event={"ID":"b1c93c32-5378-4a0d-97e9-dda679b6aad0","Type":"ContainerStarted","Data":"6f1fe807ada5e048043d843228d95feaa9f568f0fd4d0c4dd95dde8b083d06e5"}
Nov 01 21:56:48.811998 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:48.811867    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podStartSLOduration=1.81185243 podStartE2EDuration="1.81185243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:48.811478897 +0000 UTC m=+712.067658653" watchObservedRunningTime="2026-11-01 21:56:48.81185243 +0000 UTC m=+712.068032194"
Nov 01 21:56:49.208190 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.208090    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:49.209760 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.209734    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:49.209902 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.209780    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:49.338003 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.337964    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 21:56:49.342585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.342561    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.365350 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.359396    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0def36\"/\"distinct-2-w4-0def36-kserve-self-signed-certs\""
Nov 01 21:56:49.384160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.382946    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 21:56:49.517761 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.517725    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.517949 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.517765    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.517996 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.517961    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.518067 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.518039    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtvn\" (UniqueName: \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.518128 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.518111    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.518196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.518174    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.619983 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.619946    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtvn\" (UniqueName: \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620004    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620056    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620185 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620173    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620399 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620194    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620399 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620350    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620480 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620439    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620587    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.620736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.620653    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.623384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.623365    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.623690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.623673    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.632046 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.632026    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtvn\" (UniqueName: \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\") pod \"distinct-2-w4-0def36-kserve-78448f588-6rxdw\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") " pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.688965 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.688928    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:49.983544 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:49.983510    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd443e8ab_a261_40e1_9884_bbeee643fc4c.slice/crio-4f465aaa8e1f5e46a8f5f28b616300cf44ce433fc5f4f82299a7b7c6753bfb15 WatchSource:0}: Error finding container 4f465aaa8e1f5e46a8f5f28b616300cf44ce433fc5f4f82299a7b7c6753bfb15: Status 404 returned error can't find the container with id 4f465aaa8e1f5e46a8f5f28b616300cf44ce433fc5f4f82299a7b7c6753bfb15
Nov 01 21:56:49.983802 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:49.983781    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 21:56:50.207852 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:50.207819    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:50.208011 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:50.207870    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:50.804360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:50.804275    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" event={"ID":"d443e8ab-a261-40e1-9884-bbeee643fc4c","Type":"ContainerStarted","Data":"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"}
Nov 01 21:56:50.804795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:50.804370    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" event={"ID":"d443e8ab-a261-40e1-9884-bbeee643fc4c","Type":"ContainerStarted","Data":"4f465aaa8e1f5e46a8f5f28b616300cf44ce433fc5f4f82299a7b7c6753bfb15"}
Nov 01 21:56:50.825580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:50.825531    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podStartSLOduration=1.825515534 podStartE2EDuration="1.825515534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:50.823555247 +0000 UTC m=+714.079735034" watchObservedRunningTime="2026-11-01 21:56:50.825515534 +0000 UTC m=+714.081695293"
Nov 01 21:56:51.207605 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:51.207531    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:51.207765 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:51.207613    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:51.689763 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:51.689733    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:52.208330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:52.208269    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:52.208743 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:52.208357    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:53.208214 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:53.208177    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:53.208408 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:53.208234    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:54.207538 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:54.207502    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 21:56:54.207736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:54.207552    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 21:56:55.211687 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:55.211640    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:55.212105 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:55.211942    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:55.212628 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:55.212612    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 21:56:58.633906 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.633863    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 21:56:58.640194 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.640166    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.654718 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.654685    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 21:56:58.655212 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.655193    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b0b7a3\"/\"distinct-w5-b0b7a3-kserve-self-signed-certs\""
Nov 01 21:56:58.801573 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801540    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.801824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801589    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8cm\" (UniqueName: \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.801824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801734    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.801824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801792    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.801995 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801950    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.801995 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.801980    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.902989 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.902904    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.902989 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.902941    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.902989 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.902970    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903254 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903012    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8cm\" (UniqueName: \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903254 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903131    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903254 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903187    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903460 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903431    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903676 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903643    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.903789 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.903654    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.905326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.905272    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.905549 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.905533    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.915888 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.915863    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8cm\" (UniqueName: \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\") pod \"distinct-w5-b0b7a3-kserve-9956867c-czxbr\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") " pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:58.971757 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:58.971723    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:56:59.255764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.255733    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 21:56:59.257480 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:56:59.257450    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc092b7ee_3175_4c71_8df8_d69b6a6f25d3.slice/crio-a745d73141f5f756ef329a13b500106e44071bce89e0743fbfdf8eeec6a5c318 WatchSource:0}: Error finding container a745d73141f5f756ef329a13b500106e44071bce89e0743fbfdf8eeec6a5c318: Status 404 returned error can't find the container with id a745d73141f5f756ef329a13b500106e44071bce89e0743fbfdf8eeec6a5c318
Nov 01 21:56:59.689418 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.689335    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:59.701752 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.701724    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:59.890091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.890053    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" event={"ID":"c092b7ee-3175-4c71-8df8-d69b6a6f25d3","Type":"ContainerStarted","Data":"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"}
Nov 01 21:56:59.890091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.890096    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" event={"ID":"c092b7ee-3175-4c71-8df8-d69b6a6f25d3","Type":"ContainerStarted","Data":"a745d73141f5f756ef329a13b500106e44071bce89e0743fbfdf8eeec6a5c318"}
Nov 01 21:56:59.901762 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.901739    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:56:59.913462 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.913422    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podStartSLOduration=1.913409407 podStartE2EDuration="1.913409407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:56:59.91163903 +0000 UTC m=+723.167818793" watchObservedRunningTime="2026-11-01 21:56:59.913409407 +0000 UTC m=+723.169589166"
Nov 01 21:56:59.973192 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:56:59.973164    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:02.707893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.707844    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 21:57:02.713104 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.713078    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.731417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.731385    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 21:57:02.748918 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.748867    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749063 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749048    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749068    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749088    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749108    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749242    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749338 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749267    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749526 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749340    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8qxp\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.749526 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.749383    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850477 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850441    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850477 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850679 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850502    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850679 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850527    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850679 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850554    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850679 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850585    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850679 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850635    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8qxp\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.850958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850933    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851069 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.850980    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851069 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.851028    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851069 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.851060    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851187 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.851070    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851257 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.851204    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.851327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.851270    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.853391 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.853362    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.853717 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.853697    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.863116 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.863092    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:02.863241 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:02.863124    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8qxp\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:03.028878 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.028843    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 21:57:03.034755 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.034729    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.039968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.039943    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:03.047468 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.047437    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 21:57:03.155986 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.155951    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156174 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.155999    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156174 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156052    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156166    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156215    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156267    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156331    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpfx\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156649 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156425    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.156649 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.156470    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.192392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.192363    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 21:57:03.196474 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.196453    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.212245 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.212223    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 21:57:03.257451 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257426    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.257581 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.257629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257574    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.257629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257598    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.257629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257619    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.257780 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257634    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.257835 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257776    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcq7\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.257888 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257839    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.257888 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257871    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.257888 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257886    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258050 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257901    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.258050 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.257955    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258080    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258177 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258130    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258228 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258194    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258225    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258248    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.258281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258272    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.258281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258267    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258337    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpfx\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258488    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.258642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258528    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.258642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.258623    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.259961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.259938    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.260585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.260567    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.269075 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.269053    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.269663 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.269643    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpfx\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.326572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.326549    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 21:57:03.328423 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:03.328396    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86223396_2925_4ad6_ac70_34ff71808484.slice/crio-f305c3759bc3650e0e7330aba5347769c74dd0f11cfb248fbb64cc0bf27f37ba WatchSource:0}: Error finding container f305c3759bc3650e0e7330aba5347769c74dd0f11cfb248fbb64cc0bf27f37ba: Status 404 returned error can't find the container with id f305c3759bc3650e0e7330aba5347769c74dd0f11cfb248fbb64cc0bf27f37ba
Nov 01 21:57:03.330821 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.330789    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.330904 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.330850    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.330904 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.330881    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.359769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.359748    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:03.360142 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360122    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360158    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360252 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360217    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360252 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360231    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360252 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360246    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360443 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360339    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcq7\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360443 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360410    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360525    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360559    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.360700 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360684    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.361011 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360897    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.361011 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.360974    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.361168 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.361039    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.361236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.361207    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.362404 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.362385    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.362522 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.362506    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.373456 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.373431    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.373556 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.373518    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcq7\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.512160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.512119    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:03.646237 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:03.646206    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86da9fba_9c8e_49e1_8028_092beb13139b.slice/crio-3eb8142e9b5ef49684683037fb728a9b34218f1a95f2b632730363673654fe5a WatchSource:0}: Error finding container 3eb8142e9b5ef49684683037fb728a9b34218f1a95f2b632730363673654fe5a: Status 404 returned error can't find the container with id 3eb8142e9b5ef49684683037fb728a9b34218f1a95f2b632730363673654fe5a
Nov 01 21:57:03.646420 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.646373    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 21:57:03.649386 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.649348    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.649484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.649430    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.649551 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.649506    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.801726 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.801687    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 21:57:03.802086 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:03.801759    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71d4526_5b90_4e47_8778_b7ce2ebaace7.slice/crio-7d2e354e5fce24423a674c0afad5e897fa8d84ff1b16bc8a9be6b8b53fb823bf WatchSource:0}: Error finding container 7d2e354e5fce24423a674c0afad5e897fa8d84ff1b16bc8a9be6b8b53fb823bf: Status 404 returned error can't find the container with id 7d2e354e5fce24423a674c0afad5e897fa8d84ff1b16bc8a9be6b8b53fb823bf
Nov 01 21:57:03.807066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.807027    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.807645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.807616    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.807771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.807680    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:03.924950 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.924923    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" event={"ID":"e71d4526-5b90-4e47-8778-b7ce2ebaace7","Type":"ContainerStarted","Data":"7d2e354e5fce24423a674c0afad5e897fa8d84ff1b16bc8a9be6b8b53fb823bf"}
Nov 01 21:57:03.926916 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.926885    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" event={"ID":"86223396-2925-4ad6-ac70-34ff71808484","Type":"ContainerStarted","Data":"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"}
Nov 01 21:57:03.927010 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.926946    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" event={"ID":"86223396-2925-4ad6-ac70-34ff71808484","Type":"ContainerStarted","Data":"f305c3759bc3650e0e7330aba5347769c74dd0f11cfb248fbb64cc0bf27f37ba"}
Nov 01 21:57:03.933326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.929952    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" event={"ID":"86da9fba-9c8e-49e1-8028-092beb13139b","Type":"ContainerStarted","Data":"56eb684e1be9c4dfcf8253cdac6b57c6d2b2b9dac3a16d7260ca60f16599ea29"}
Nov 01 21:57:03.933326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.930004    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" event={"ID":"86da9fba-9c8e-49e1-8028-092beb13139b","Type":"ContainerStarted","Data":"3eb8142e9b5ef49684683037fb728a9b34218f1a95f2b632730363673654fe5a"}
Nov 01 21:57:03.989227 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.989180    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podStartSLOduration=1.98916662 podStartE2EDuration="1.98916662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:03.960045576 +0000 UTC m=+727.216225340" watchObservedRunningTime="2026-11-01 21:57:03.98916662 +0000 UTC m=+727.245346421"
Nov 01 21:57:03.995407 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:03.995359    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podStartSLOduration=1.995341979 podStartE2EDuration="1.995341979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:03.992604222 +0000 UTC m=+727.248783989" watchObservedRunningTime="2026-11-01 21:57:03.995341979 +0000 UTC m=+727.251521744"
Nov 01 21:57:04.040781 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.040750    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:04.042056 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.042036    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:04.042207 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.042189    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:04.367322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.366742    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:04.393330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.390601    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:04.393330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.390679    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:04.944530 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:57:04.944493    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:57:05.003322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:04.987812    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" event={"ID":"e71d4526-5b90-4e47-8778-b7ce2ebaace7","Type":"ContainerStarted","Data":"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"}
Nov 01 21:57:05.036470 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.036217    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podStartSLOduration=2.036198561 podStartE2EDuration="2.036198561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:05.032864254 +0000 UTC m=+728.289044022" watchObservedRunningTime="2026-11-01 21:57:05.036198561 +0000 UTC m=+728.292378324"
Nov 01 21:57:05.041706 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.041666    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:05.041858 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.041732    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:05.364322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.363649    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:05.364322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.363738    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:05.514321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.513362    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:05.514321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.514147    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:05.514321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:05.514200    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:06.040865 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.040823    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:06.041374 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.040885    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:06.365376 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.364843    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:06.365376 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.364903    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:06.520321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.520189    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:06.524318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:06.520360    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:07.068324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.058422    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:07.068324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.058491    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:07.401909 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.401824    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:07.427428 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.427386    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:07.544126 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.542424    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:07.544126 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:07.542485    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:08.076379 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.040804    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:08.076379 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.040871    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:08.369377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.368694    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:08.369377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.368760    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:08.514324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.513706    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:08.514324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.513766    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:08.975511 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:08.974626    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:09.031930 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.031903    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:09.041291 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.041257    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:09.041437 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.041328    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:09.235834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.235739    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:09.362320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.360338    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 01 21:57:09.362320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.360399    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 01 21:57:09.515378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.514619    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:09.515378 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:09.514666    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:10.040805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.040772    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 01 21:57:10.041013 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.040832    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 01 21:57:10.513568 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.513531    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 21:57:10.514027 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.513595    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 21:57:10.609119 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.609088    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:10.610122 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.610041    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:10.610805 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:10.610780    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 21:57:11.051276 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.051242    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:11.051760 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.051737    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:11.053619 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.053595    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 21:57:11.518487 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.518449    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:11.519177 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.519158    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:11.520018 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:11.519996    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 21:57:16.861939 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.861909    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 21:57:16.865831 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.865811    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.872161 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.872139    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 21:57:16.872385 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.872368    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8pxdd\""
Nov 01 21:57:16.872484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.872403    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 21:57:16.872484 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.872425    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 21:57:16.874472 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.874214    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 21:57:16.976049 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976014    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjn7n\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976052    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976099    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976197 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976185    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976345 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976206    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976345 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976337    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:16.976500 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:16.976486    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.077855 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.077824    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjn7n\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.077855 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.077859    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.078039 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.077876    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.078039 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.077984    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.078039 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.078031    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.078133 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.078077    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.078327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.078272    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.079034 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.079006    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.080337 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.080291    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.080473 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.080455    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.080595 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.080574    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.080650 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.080578    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.089945 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.089925    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.090044 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.090030    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjn7n\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dsdmp\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.180849 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.180772    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:17.466893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:17.466863    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 21:57:17.467057 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:17.467025    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31852f01_d286_4d4d_b3cd_1819d347bf60.slice/crio-c13405c57dc5b28c970a470dcb17706acf3850cf9e79cd29a0c70ad203aa627c WatchSource:0}: Error finding container c13405c57dc5b28c970a470dcb17706acf3850cf9e79cd29a0c70ad203aa627c: Status 404 returned error can't find the container with id c13405c57dc5b28c970a470dcb17706acf3850cf9e79cd29a0c70ad203aa627c
Nov 01 21:57:18.208873 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:18.208839    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" event={"ID":"31852f01-d286-4d4d-b3cd-1819d347bf60","Type":"ContainerStarted","Data":"c13405c57dc5b28c970a470dcb17706acf3850cf9e79cd29a0c70ad203aa627c"}
Nov 01 21:57:20.633647 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:20.633618    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:20.633939 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:20.633679    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:21.251489 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:21.251453    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" event={"ID":"31852f01-d286-4d4d-b3cd-1819d347bf60","Type":"ContainerStarted","Data":"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"}
Nov 01 21:57:21.252036 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:21.252005    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:21.254860 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:21.254835    5845 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dsdmp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 21:57:21.254989 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:21.254881    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 21:57:21.284150 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:21.284099    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podStartSLOduration=2.12079372 podStartE2EDuration="5.284085848s" totalImagesPullingTime="3.163292128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:21.28180379 +0000 UTC m=+744.537983574" watchObservedRunningTime="2026-11-01 21:57:21.284085848 +0000 UTC m=+744.540265610"
Nov 01 21:57:22.270388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:22.266643    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 21:57:34.816438 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:57:34.816397    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:57:47.908024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:47.907989    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 21:57:47.914502 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:47.914474    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:47.933354 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:47.933324    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 21:57:48.040346 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040270    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040590 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040572    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040613    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040690 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040640    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4x6\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040804 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040694    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040804 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040743    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.040935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040904    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.041006 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.040991    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.041268 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.041252    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.088742 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.088702    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 21:57:48.093196 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.093168    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.113736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.113705    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 21:57:48.142921 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.142875    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.142939    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.142960    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4x6\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.142989    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143039    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143073    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143085 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143092    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143151    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143179    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.143737 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.143688    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.144772 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.144741    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.145175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.145154    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.145362 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.145342    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.145570 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.145541    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.147583 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.147539    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.153082 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.153054    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.157555 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.157529    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.158137 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.158085    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4x6\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.235492 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.235464    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:48.244203 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244170    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244219    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244432 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244370    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrmv\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244447    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244564 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244535    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244699 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244681    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244745 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244719    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.244941 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.244899    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.245060 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.245013    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.286891 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.285708    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 21:57:48.291275 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.291247    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.313629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.313509    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.346934    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347006    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347072    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347107    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347176    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrmv\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347249    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347290    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347423    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.347453    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.348258    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.348579    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.348821    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.349062    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.350261    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.356353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.354726    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.362764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.362739    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.368086 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.368053    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrmv\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.371953 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.371906    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\") pod \"e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") " pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.416066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.415974    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:48.449022 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.448986    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449107    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsth\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449160 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449145    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449239 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449206    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449276 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449248    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449281    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449314    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449495 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449481    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.449535 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.449507    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.546284 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.546258    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 21:57:48.548857 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:48.548829    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeae6af7_38cc_4619_a61f_fe4eb01608d0.slice/crio-247781f5ff9e2e5cb5a4989d32d1caf5fdf541bf91b65b78d490d71dc5e8852c WatchSource:0}: Error finding container 247781f5ff9e2e5cb5a4989d32d1caf5fdf541bf91b65b78d490d71dc5e8852c: Status 404 returned error can't find the container with id 247781f5ff9e2e5cb5a4989d32d1caf5fdf541bf91b65b78d490d71dc5e8852c
Nov 01 21:57:48.550689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550656    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.550830 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550729    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsth\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.550830 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550773    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.550954 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550851    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.550954 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550913    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550961    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.550985    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551055    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551215 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551061    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551215 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551080    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551407 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551383    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551661 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551635    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.551774 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.551751    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.553071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.553033    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:48.553172 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.553116    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:48.553237 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.553184    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:48.553872 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.553850    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.554454 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.554316    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.556685 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.556661    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.569337 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.569310    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.569453 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.569411    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsth\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\") pod \"e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") " pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.636711 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.636683    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:48.724251 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:57:48.724214    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c3afe8_6835_4478_92fe_a8a0afddfed5.slice/crio-afce0d20ff9adf1d6b8c7a186698397b9789b8a94f7356081f51b252637d029a WatchSource:0}: Error finding container afce0d20ff9adf1d6b8c7a186698397b9789b8a94f7356081f51b252637d029a: Status 404 returned error can't find the container with id afce0d20ff9adf1d6b8c7a186698397b9789b8a94f7356081f51b252637d029a
Nov 01 21:57:48.728062 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.728030    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 21:57:48.734598 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.734566    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:48.734713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.734644    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:48.734713 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:48.734687    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:49.156345 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.156285    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:49.156771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.156396    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:49.156771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.156442    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:57:49.180395 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.178743    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 21:57:49.949321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.943956    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" event={"ID":"f2c3afe8-6835-4478-92fe-a8a0afddfed5","Type":"ContainerStarted","Data":"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"}
Nov 01 21:57:49.949321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.944045    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" event={"ID":"f2c3afe8-6835-4478-92fe-a8a0afddfed5","Type":"ContainerStarted","Data":"afce0d20ff9adf1d6b8c7a186698397b9789b8a94f7356081f51b252637d029a"}
Nov 01 21:57:49.971317 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:49.970866    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" event={"ID":"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587","Type":"ContainerStarted","Data":"a1968cef79d886219ff7b1701e9e6b3d2f877ccddf2b48d839ac4acfdea3cfdb"}
Nov 01 21:57:50.017906 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.017835    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podStartSLOduration=2.0178154 podStartE2EDuration="2.0178154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:50.016089094 +0000 UTC m=+773.272268858" watchObservedRunningTime="2026-11-01 21:57:50.0178154 +0000 UTC m=+773.273995164"
Nov 01 21:57:50.053321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.028053    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" event={"ID":"beae6af7-38cc-4619-a61f-fe4eb01608d0","Type":"ContainerStarted","Data":"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"}
Nov 01 21:57:50.053321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.028118    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" event={"ID":"beae6af7-38cc-4619-a61f-fe4eb01608d0","Type":"ContainerStarted","Data":"247781f5ff9e2e5cb5a4989d32d1caf5fdf541bf91b65b78d490d71dc5e8852c"}
Nov 01 21:57:50.054769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.054702    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podStartSLOduration=2.054681038 podStartE2EDuration="2.054681038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:50.046842942 +0000 UTC m=+773.303022706" watchObservedRunningTime="2026-11-01 21:57:50.054681038 +0000 UTC m=+773.310860799"
Nov 01 21:57:50.237971 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.237939    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:57:50.238565 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.238542    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:50.238709 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.238687    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:50.432322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.426575    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:57:50.432322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.426690    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:50.432322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.426728    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:50.660427 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.656637    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:57:50.660427 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.658794    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:50.660427 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.658844    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:50.682356 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.673196    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 21:57:50.706553 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.691472    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:50.772824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.772781    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:57:50.999398 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:50.912893    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.005041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.002624    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.005041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.003697    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.005041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.004395    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.118208 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.117422    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.175319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.155495    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.355485 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.354201    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:57:51.355485 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.354535    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.378167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.377682    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" event={"ID":"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587","Type":"ContainerStarted","Data":"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"}
Nov 01 21:57:51.401334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.399415    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:51.401334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.399479    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:51.422523 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.422489    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:51.422826 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.422778    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:51.467074 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.459434    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 01 21:57:51.498624 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.498582    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.642379 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.642276    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:51.642725 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.642698    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:51.702585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.702537    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:51.725331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.719438    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:57:51.768992 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.755853    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podStartSLOduration=4.755832674 podStartE2EDuration="4.755832674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:57:50.086587777 +0000 UTC m=+773.342767538" watchObservedRunningTime="2026-11-01 21:57:51.755832674 +0000 UTC m=+775.012012437"
Nov 01 21:57:51.782468 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.782422    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:57:51.909775 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.909686    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:57:51.921993 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.921394    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 21:57:51.922279 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:51.922255    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:57:52.010411 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.009680    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:52.041381 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.036670    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:57:52.044908 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.041684    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:57:52.044908 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.041751    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:57:52.044908 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.044263    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 21:57:52.046351 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.046275    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:57:52.133355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.124386    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:57:52.133355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.125117    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:57:52.191499 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.191402    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 21:57:52.192352 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.192326    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:52.288321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.267022    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:52.288321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.267084    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:52.345364 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.343435    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:52.371321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.365384    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:52.371321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.365787    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:57:52.371321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.366082    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:57:52.371321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.366138    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:57:52.503411 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.430938    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:52.503411 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.430987    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:52.748320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.716211    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.003381 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.932416    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.036774 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.939986    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.040324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.037098    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:57:53.057323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:52.950419    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:53.057323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.056879    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:53.068148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.057995    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.068148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.058063    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.068148 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.064949    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.214355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.198837    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:57:53.214355 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.201615    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:53.269187 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.269082    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:53.269488 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.269456    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:53.373633 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.373585    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:57:53.374377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.374342    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 21:57:53.437325 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.427825    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:53.500824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.500548    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:53.707866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.707740    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:53.707866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.707822    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:53.741327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.739857    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.006781 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:53.940539    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 01 21:57:54.043520 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.034735    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.056425 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.056354    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:57:54.057071 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.057024    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 21:57:54.074775 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.074741    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.075927 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.075892    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.214890 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.214840    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.221809 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.217450    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:57:54.236664 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.236627    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:54.236935 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.236893    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:54.463697 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.463579    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:57:54.464799 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.463844    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 21:57:54.515342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.515289    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:54.515613 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.515584    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:54.655872 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.655835    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:54.656170 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.656128    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:54.741569 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.741513    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:54.855831 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:54.855791    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:57:55.054494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.054397    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:57:55.068423 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.068381    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:57:55.069000 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.068657    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:55.082707 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.082670    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:57:55.220344 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.205687    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:55.237541 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.235498    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 21:57:55.273417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.272286    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:55.273417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.272368    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:55.418844 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.418759    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:57:55.470334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.441366    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:55.470334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.441409    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:55.470334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.443829    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:57:55.470334 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.468083    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:57:55.535349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.527388    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:55.535349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.533778    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:57:55.623820 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.623787    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:57:55.624673 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.624648    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:57:55.637634 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.637607    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:55.637836 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.637813    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:55.768213 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.768182    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:57:55.963618 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:55.963577    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:57:56.027522 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.027433    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:57:56.079531 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.079490    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:57:56.247444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.247403    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:56.247675 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.247645    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:56.346777 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.346691    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:57:56.427431 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.427394    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:56.432405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.432365    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:56.638386 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.638282    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:56.638926 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:56.638890    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:57.236291 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.236257    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:57.236617 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.236580    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:57.424323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.422787    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:57.424323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.422849    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:57.637182 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.637107    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:57.637408 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:57.637384    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:58.236315 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.236058    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:58.236315 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.236116    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:58.419444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.419405    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:58.419633 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.419565    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:58.637718 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.637625    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:58.637718 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:58.637677    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:57:59.235914 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.235878    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 21:57:59.236119 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.235936    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 21:57:59.416687 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.416653    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 21:57:59.417068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.416719    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 21:57:59.637675 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.637587    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 01 21:57:59.637675 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:57:59.637647    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 01 21:58:00.242525 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.242489    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:58:00.243024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.242967    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:58:00.243809 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.243759    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 21:58:00.508443 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.508364    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:58:00.509783 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.509760    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:58:00.510863 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.510840    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 21:58:00.748360 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.748322    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:58:00.748694 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.748670    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:58:00.749756 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:00.749730    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 21:58:02.918217 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:02.918185    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 21:58:02.926494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:02.926475    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:02.935425 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:02.935404    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 21:58:03.001377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001351    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001508 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001389    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001508 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001473    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001578 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001516    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001634 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001614    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001694 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001680    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001730 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001723    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001787 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001775    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxmc\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.001821 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.001803    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103266 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103238    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103278    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103431    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103528 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103472    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103577 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103561    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103617 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103602    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103686 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103665    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103759 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103739    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103799 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103666    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103847 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103816    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxmc\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103885 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103864    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.103941 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.103910    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.104217 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.104196    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.104359 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.104336    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.105733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.105712    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.105993 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.105976    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.117257 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.116945    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxmc\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.119409 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.119385    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\") pod \"e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") " pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.242651 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.242623    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:03.527097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.527069    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 21:58:03.528699 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:58:03.528663    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e40483_a711_464c_b026_15e3d5b938e6.slice/crio-1eec66a9edead89e7445a587414d94768cfce044c33a0f8bb75162aabb1b0b86 WatchSource:0}: Error finding container 1eec66a9edead89e7445a587414d94768cfce044c33a0f8bb75162aabb1b0b86: Status 404 returned error can't find the container with id 1eec66a9edead89e7445a587414d94768cfce044c33a0f8bb75162aabb1b0b86
Nov 01 21:58:03.531263 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.531228    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:03.531399 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.531310    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:03.531399 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:03.531342    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:04.388834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:04.388794    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" event={"ID":"c5e40483-a711-464c-b026-15e3d5b938e6","Type":"ContainerStarted","Data":"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"}
Nov 01 21:58:04.389282 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:04.388860    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" event={"ID":"c5e40483-a711-464c-b026-15e3d5b938e6","Type":"ContainerStarted","Data":"1eec66a9edead89e7445a587414d94768cfce044c33a0f8bb75162aabb1b0b86"}
Nov 01 21:58:04.418021 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:04.417969    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podStartSLOduration=2.417955964 podStartE2EDuration="2.417955964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:04.414851365 +0000 UTC m=+787.671031140" watchObservedRunningTime="2026-11-01 21:58:04.417955964 +0000 UTC m=+787.674135726"
Nov 01 21:58:04.832066 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:58:04.832029    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:58:05.243405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:05.243372    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:05.244952 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:05.244912    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:05.245173 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:05.245147    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:06.244057 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:06.243656    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:06.244057 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:06.243739    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:07.243866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:07.243834    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:07.244036 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:07.243883    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:08.255350 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:08.249958    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:08.255350 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:08.250024    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:09.243902 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:09.243872    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:09.244078 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:09.243919    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:10.243674 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:10.243639    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 21:58:10.244079 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:10.243707    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 21:58:11.271275 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:11.271244    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:11.271674 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:11.271594    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:11.272580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:11.272557    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 21:58:17.881929 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.881900    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 21:58:17.887764 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.887740    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.900285 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.900257    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 21:58:17.973336 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973269    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973369    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973427    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973489    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7pq\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973546    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973700    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973742    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973790 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973776    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:17.973939 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:17.973800    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.075400 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.075341    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.075838 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.075695    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.076098 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.076031    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.076210 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.076101    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.076503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.076444    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7pq\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.076739 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.076704    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.077041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.076984    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.077274 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.077215    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.082377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.077417    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.082377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.079087    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.082377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.079530    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.082377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.080971    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.083198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.082692    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.083967 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.083942    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.089357 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.087805    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.100237 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.100206    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7pq\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.102328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.102288    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.104924 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.104896    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\") pod \"e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") " pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.209289 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.209224    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:18.214209 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.214183    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 21:58:18.219242 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.218882    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.234538 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.234515    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.286765    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.286895    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6k4\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.286985    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287054    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287130    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287159    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287202    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.287349 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287242    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.288220 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.287912    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389215 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389183    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389252    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389328    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389357    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389397    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389633 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389438    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389729 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389670    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389875 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389753    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.389875 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389769    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.390015 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389949    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.390015 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.389989    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.390117 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.390018    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6k4\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.390117 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.390075    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.390222 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.390115    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.392029 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.391999    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.392308 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.392283    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.405795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.405776    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.407236 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.407219    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6k4\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\") pod \"e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") " pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.506710 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.506688    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 21:58:18.507266 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:58:18.507239    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d537ecc_748c_402e_9f5d_dd6855d93a88.slice/crio-d758157ffb047469e2ba19843744a2910802454ee3706321ea89ee0b1b864c21 WatchSource:0}: Error finding container d758157ffb047469e2ba19843744a2910802454ee3706321ea89ee0b1b864c21: Status 404 returned error can't find the container with id d758157ffb047469e2ba19843744a2910802454ee3706321ea89ee0b1b864c21
Nov 01 21:58:18.509645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.509613    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:18.509724 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.509676    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:18.509724 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.509709    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:18.568176 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.568152    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:18.865958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.865929    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 21:58:18.866568 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:58:18.866542    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7bbaad_d051_4782_a918_386d7f69bd6d.slice/crio-c70fc7b1fe6a0eda448f60cda77446a0fbb178d3e908cb71125b2964aa5c26b1 WatchSource:0}: Error finding container c70fc7b1fe6a0eda448f60cda77446a0fbb178d3e908cb71125b2964aa5c26b1: Status 404 returned error can't find the container with id c70fc7b1fe6a0eda448f60cda77446a0fbb178d3e908cb71125b2964aa5c26b1
Nov 01 21:58:18.869080 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.869053    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:18.869130 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.869112    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:18.869168 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:18.869141    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:58:19.505562 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.505518    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" event={"ID":"9d537ecc-748c-402e-9f5d-dd6855d93a88","Type":"ContainerStarted","Data":"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"}
Nov 01 21:58:19.506034 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.505578    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" event={"ID":"9d537ecc-748c-402e-9f5d-dd6855d93a88","Type":"ContainerStarted","Data":"d758157ffb047469e2ba19843744a2910802454ee3706321ea89ee0b1b864c21"}
Nov 01 21:58:19.507191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.507168    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" event={"ID":"ae7bbaad-d051-4782-a918-386d7f69bd6d","Type":"ContainerStarted","Data":"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"}
Nov 01 21:58:19.507327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.507206    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" event={"ID":"ae7bbaad-d051-4782-a918-386d7f69bd6d","Type":"ContainerStarted","Data":"c70fc7b1fe6a0eda448f60cda77446a0fbb178d3e908cb71125b2964aa5c26b1"}
Nov 01 21:58:19.531751 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.531689    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podStartSLOduration=2.531669099 podStartE2EDuration="2.531669099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.528743801 +0000 UTC m=+802.784923608" watchObservedRunningTime="2026-11-01 21:58:19.531669099 +0000 UTC m=+802.787848860"
Nov 01 21:58:19.564331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.562436    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podStartSLOduration=1.562417698 podStartE2EDuration="1.562417698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:58:19.560825859 +0000 UTC m=+802.817005660" watchObservedRunningTime="2026-11-01 21:58:19.562417698 +0000 UTC m=+802.818597461"
Nov 01 21:58:19.577580 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.577540    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:19.582086 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.582053    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:19.582230 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:19.582113    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:20.210796 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:20.210755    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:20.212533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:20.212438    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:20.212533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:20.212510    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:20.569321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:20.568711    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:20.569321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:20.568774    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:21.216461 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:21.216388    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:21.216653 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:21.216469    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:21.583354 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:21.579536    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:21.583354 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:21.579609    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:22.209595 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:22.209559    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:22.209772 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:22.209610    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:22.569361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:22.568882    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:22.569361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:22.568947    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:23.210893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:23.210554    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:23.210893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:23.210600    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:23.581732 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:23.581694    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:23.582065 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:23.582034    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:24.311320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:24.286534    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:24.311320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:24.286596    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:24.569812 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:24.569726    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:24.570062 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:24.570033    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:25.211326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.210623    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:25.211326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.210682    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:25.500321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.495562    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:25.573331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.572896    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:25.573331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.572975    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:25.738467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:25.738380    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.089320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.089055    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.093319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.092314    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.093319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.092416    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.093319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.092593    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 21:58:26.093319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.092659    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:58:26.213412 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.213377    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:26.213668 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.213639    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:26.221962 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.221919    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 21:58:26.237628 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.237501    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:58:26.382379 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.380794    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.498445 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.498397    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.522323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.499367    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:58:26.620463 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.620428    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:26.620697 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.620671    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:26.687776 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.687658    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:58:26.687776 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.687734    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:26.766321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.766033    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 21:58:26.776366 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:26.766861    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:58:27.140455 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.140406    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.193476 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.193432    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:58:27.193839 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.193738    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.223405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.218439    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.223405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.218582    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 21:58:27.223405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.218612    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 21:58:27.223405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.220867    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 01 21:58:27.223405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.221982    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 21:58:27.262166 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.262106    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:58:27.263110 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.263076    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.270663 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.270619    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.307823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.307764    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Nov 01 21:58:27.486235 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.486192    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:27.522331 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.521717    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:58:27.575644 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.575588    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 01 21:58:27.575831 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.575661    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 01 21:58:27.657320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.636671    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:58:27.712318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712270    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:58:27.712521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712373    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:58:27.712521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712450    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:58:27.712521 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712505    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:58:27.712697 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712559    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:58:27.712697 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712615    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:58:27.712697 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712671    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:58:27.712845 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:27.712730    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:58:28.169320 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.141002    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.221963 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.221907    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:58:28.222344 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.222316    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.223279 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.222347    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:58:28.272498 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.272446    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.295008 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.294963    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:58:28.295629 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.295587    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 21:58:28.412983 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.412950    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:58:28.483478 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.476967    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.560366 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.552395    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.580227 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.561814    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:28.583045 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.582990    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:58:28.614081 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.614057    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:58:28.685416 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.684990    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:58:28.714567 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.714021    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:58:28.717400 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.715326    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:58:28.717400 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.717179    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:58:28.722342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.722076    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:58:28.736840 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.736148    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:58:28.819771 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.819737    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:28.821129 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.821098    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:28.825342 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:28.822026    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 21:58:29.071710 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:29.071609    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:29.071928 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:29.071907    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:29.072711 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:29.072691    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 21:58:34.816260 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:58:34.816208    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:58:56.239972 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.239440    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:58:56.283442 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.283394    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:56.426318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.424562    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:58:56.458426 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.452233    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:56.597376 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.596403    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:56.736322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.732402    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:56.910919 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:56.761381    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.209766 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.209655    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.588352    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.602674    5845 patch_prober.go:29] interesting pod/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.603230    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.646422    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.687741    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.688180    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.752523    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.754190    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.784191    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.784609    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.810395    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.810530    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.810585    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Nov 01 21:58:57.842321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.810619    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:58:57.978328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:57.978266    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:58:58.241094 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:58:58.241057    5845 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.314s"
Nov 01 21:58:58.241417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.241402    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:58:58.241600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.241581    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:58:58.241736 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.241726    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:58:58.241867 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.241857    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:58:58.241997 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.241987    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:58:58.242123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.242113    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:58:58.242242 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.242232    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:58:58.242391 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.242377    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:58:58.291175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.282121    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:58:58.291175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.283154    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:58:58.291175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.283224    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:58:58.291175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.286032    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:58:58.291175 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.286078    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:58:58.313335 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:58:58.311833    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:59:01.530157 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.530123    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:01.535922 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.535896    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.545068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.544635    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-13364262-openshift-default-dockercfg-hk9r6\""
Nov 01 21:59:01.546607 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.546580    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:01.559072 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559045    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559218 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559085    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559218 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559194    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559239    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559393 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559373    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559438 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559405    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559595 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559573    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrzk\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559824 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559806    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.559868 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.559841    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661469    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661503 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661511    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661783 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661695    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrzk\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661930 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661912    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661986 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661937    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.661986 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.661948    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662023    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662091 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662070    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662200 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662115    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662200 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662144    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662200 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662184    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662465 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662441    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662548 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662457    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.662795 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.662759    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.664479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.664453    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.664666 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.664648    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.681527 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.681491    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.682137 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.682106    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrzk\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\") pod \"e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") " pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:01.857119 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:01.857024    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:02.150287 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.150257    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:02.151711 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:02.151665    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e7747b_833b_4c5b_9edb_871a637e2458.slice/crio-140b38adfc18c9b9547ffa3e1ff57e9cbdf5595b55dc22e969c9ce6dabf687a0 WatchSource:0}: Error finding container 140b38adfc18c9b9547ffa3e1ff57e9cbdf5595b55dc22e969c9ce6dabf687a0: Status 404 returned error can't find the container with id 140b38adfc18c9b9547ffa3e1ff57e9cbdf5595b55dc22e969c9ce6dabf687a0
Nov 01 21:59:02.155683 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.155646    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:02.155774 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.155737    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:02.155810 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.155781    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:02.273968 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.273932    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerStarted","Data":"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"}
Nov 01 21:59:02.274154 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.273998    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerStarted","Data":"140b38adfc18c9b9547ffa3e1ff57e9cbdf5595b55dc22e969c9ce6dabf687a0"}
Nov 01 21:59:02.299993 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.299925    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podStartSLOduration=1.299906583 podStartE2EDuration="1.299906583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:02.296821012 +0000 UTC m=+845.553000802" watchObservedRunningTime="2026-11-01 21:59:02.299906583 +0000 UTC m=+845.556086348"
Nov 01 21:59:02.905353 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:02.885370    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:03.863284 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:03.863248    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:04.302264 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:04.302234    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:04.303761 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:04.303727    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:04.821556 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:04.821516    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:59:12.460208 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.460169    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:12.466007 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.465981    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.475285 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.475247    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-13364262-serving-cert\""
Nov 01 21:59:12.476791 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.476763    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:12.490797 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.490761    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvzg\" (UniqueName: \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.490958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.490919    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.491013 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.490969    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.491103 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.491086    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.592555 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.592521    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.592555 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.592583    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.592850 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.592703    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.592850 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.592836    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvzg\" (UniqueName: \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.593283 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:12.593180    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-13364262-serving-cert: secret "maas-api-e2e-ait-13364262-serving-cert" not found
Nov 01 21:59:12.593283 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:12.593275    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls podName:cfa10098-ec06-46cf-a7bb-3213eddca65d nodeName:}" failed. No retries permitted until 2026-11-01 21:59:13.093252463 +0000 UTC m=+856.349432219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls") pod "maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" (UID: "cfa10098-ec06-46cf-a7bb-3213eddca65d") : secret "maas-api-e2e-ait-13364262-serving-cert" not found
Nov 01 21:59:12.593991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.593966    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.598623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.598598    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:12.606286 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:12.606258    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvzg\" (UniqueName: \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:13.100480 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.100437    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:13.135324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.132021    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") pod \"maas-api-e2e-ait-13364262-777d6b4cd-c8nnx\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:13.390855 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.390774    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:13.538231 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.538195    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:13.543526 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.543497    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.557256 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.556940    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-681db477-serving-cert\""
Nov 01 21:59:13.563393 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.563352    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:13.609438 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.609395    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.609742 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.609717    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.609785 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.609755    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27zg\" (UniqueName: \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.609823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.609784    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.707053 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.707014    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:13.708455 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:13.708418    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa10098_ec06_46cf_a7bb_3213eddca65d.slice/crio-68a77635e6c775692d9134fde38cdea55e541716784c4878e86773031d969daa WatchSource:0}: Error finding container 68a77635e6c775692d9134fde38cdea55e541716784c4878e86773031d969daa: Status 404 returned error can't find the container with id 68a77635e6c775692d9134fde38cdea55e541716784c4878e86773031d969daa
Nov 01 21:59:13.709950 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.709917    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 21:59:13.711641 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.711612    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.711757 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.711677    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27zg\" (UniqueName: \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.711757 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.711726    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.711871 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.711784    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.713419 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.713373    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.714739 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.714717    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.715123 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.715106    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.726263 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.726234    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27zg\" (UniqueName: \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\") pod \"maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:13.868997 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:13.868959    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:14.224461 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:14.218003    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:14.225536 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:14.225485    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912476b8_9e30_4f52_9019_3e7daba96f50.slice/crio-b8b771fc6cd16c2f1b1db399f608b517b6d7df5f2232d52c175438e0723c6342 WatchSource:0}: Error finding container b8b771fc6cd16c2f1b1db399f608b517b6d7df5f2232d52c175438e0723c6342: Status 404 returned error can't find the container with id b8b771fc6cd16c2f1b1db399f608b517b6d7df5f2232d52c175438e0723c6342
Nov 01 21:59:14.378646 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:14.378594    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" event={"ID":"912476b8-9e30-4f52-9019-3e7daba96f50","Type":"ContainerStarted","Data":"b8b771fc6cd16c2f1b1db399f608b517b6d7df5f2232d52c175438e0723c6342"}
Nov 01 21:59:14.381978 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:14.381933    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" event={"ID":"cfa10098-ec06-46cf-a7bb-3213eddca65d","Type":"ContainerStarted","Data":"ed69c9ccce57c8ff1c579e64af5be1b265317f68dd4a80781f808f217b232b04"}
Nov 01 21:59:14.382152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:14.382000    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" event={"ID":"cfa10098-ec06-46cf-a7bb-3213eddca65d","Type":"ContainerStarted","Data":"68a77635e6c775692d9134fde38cdea55e541716784c4878e86773031d969daa"}
Nov 01 21:59:14.382152 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:14.382038    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:15.390685 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:15.390637    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" event={"ID":"912476b8-9e30-4f52-9019-3e7daba96f50","Type":"ContainerStarted","Data":"430a57a2a13ae96de44087fb18bf32d5d75a352f7d8264ac2de963358e75709d"}
Nov 01 21:59:15.391114 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:15.390816    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:15.413280 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:15.413229    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" podStartSLOduration=2.00389178 podStartE2EDuration="2.413215411s" totalImagesPullingTime="409.323631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:15.411368167 +0000 UTC m=+858.667547983" watchObservedRunningTime="2026-11-01 21:59:15.413215411 +0000 UTC m=+858.669395197"
Nov 01 21:59:15.414965 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:15.414922    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" podStartSLOduration=2.8511630930000003 podStartE2EDuration="3.414912313s" totalImagesPullingTime="563.74922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:14.408388637 +0000 UTC m=+857.664568405" watchObservedRunningTime="2026-11-01 21:59:15.414912313 +0000 UTC m=+858.671092111"
Nov 01 21:59:17.959155 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:17.959118    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 21:59:17.963714 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:17.963683    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:17.979142 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:17.979070    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 21:59:18.005441 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.003209    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 21:59:18.009283 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.008189    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.025859 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.025830    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 21:59:18.095955 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.095921    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096110 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.095967    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096110 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096059    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096191 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096110    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096226 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096205    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096260 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096231    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096293 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096266    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096417 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096403    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096453 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096429    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096486 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096465    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096569 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096549    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpt97\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096581    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096657 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096634    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sl5l\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096689 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096682    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.096723 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096710    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096756 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096730    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096809 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096797    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.096866 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.096855    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.198789 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198756    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198797    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpt97\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198818    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198847    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sl5l\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198891    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198927    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.198974 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.198958    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199000    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199081    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199144    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199185    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199221    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199236    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199264    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199313 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199287    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199550    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.199643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199562    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199602    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199812 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199648    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199812 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199747    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199812 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199780    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.199812 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.199795    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.200094 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200070    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.200167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200094    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.200208 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200185    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.200433 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200413    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.200596 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200562    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.200674 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.200589    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.201961 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.201940    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.202356 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.202286    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.202467 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.202450    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.202553 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.202535    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.210642 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.210590    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpt97\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.210755 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.210737    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sl5l\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.210904 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.210862    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.211066 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.211050    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.286567 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.286534    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:18.327479 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.327450    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:18.585543 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.585509    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 21:59:18.586523 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:18.586499    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cc4be8_1e86_447d_b363_da9a4a2c3c6b.slice/crio-f0204fb144afadb879eaddea13d7416ac651e285bddb8feef2d7a3397be4fe40 WatchSource:0}: Error finding container f0204fb144afadb879eaddea13d7416ac651e285bddb8feef2d7a3397be4fe40: Status 404 returned error can't find the container with id f0204fb144afadb879eaddea13d7416ac651e285bddb8feef2d7a3397be4fe40
Nov 01 21:59:18.589444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.589404    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:18.589545 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.589488    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:18.589545 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.589531    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:18.651202 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.651178    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 21:59:18.653543 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:18.653519    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23715e0a_24d9_4d1f_91c5_a02cf518c781.slice/crio-9fad217a949bf95d68ee953e71460333cdfd76eda5309c5570a4c970896f0800 WatchSource:0}: Error finding container 9fad217a949bf95d68ee953e71460333cdfd76eda5309c5570a4c970896f0800: Status 404 returned error can't find the container with id 9fad217a949bf95d68ee953e71460333cdfd76eda5309c5570a4c970896f0800
Nov 01 21:59:18.656457 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.656421    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:18.656572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.656492    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:18.656572 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:18.656522    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:19.469030 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.468992    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" event={"ID":"23715e0a-24d9-4d1f-91c5-a02cf518c781","Type":"ContainerStarted","Data":"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"}
Nov 01 21:59:19.469481 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.469058    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" event={"ID":"23715e0a-24d9-4d1f-91c5-a02cf518c781","Type":"ContainerStarted","Data":"9fad217a949bf95d68ee953e71460333cdfd76eda5309c5570a4c970896f0800"}
Nov 01 21:59:19.470654 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.470627    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" event={"ID":"30cc4be8-1e86-447d-b363-da9a4a2c3c6b","Type":"ContainerStarted","Data":"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"}
Nov 01 21:59:19.470769 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.470674    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" event={"ID":"30cc4be8-1e86-447d-b363-da9a4a2c3c6b","Type":"ContainerStarted","Data":"f0204fb144afadb879eaddea13d7416ac651e285bddb8feef2d7a3397be4fe40"}
Nov 01 21:59:19.493562 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.493515    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podStartSLOduration=2.493503707 podStartE2EDuration="2.493503707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.492350762 +0000 UTC m=+862.748530526" watchObservedRunningTime="2026-11-01 21:59:19.493503707 +0000 UTC m=+862.749683469"
Nov 01 21:59:19.515861 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:19.515808    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podStartSLOduration=2.515792145 podStartE2EDuration="2.515792145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:19.514414362 +0000 UTC m=+862.770594129" watchObservedRunningTime="2026-11-01 21:59:19.515792145 +0000 UTC m=+862.771971908"
Nov 01 21:59:20.287949 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.287875    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:20.297377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.290963    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:20.297377 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.291021    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:20.327756 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.327719    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:20.329111 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.328999    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:20.329111 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.329073    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:20.402184 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:20.402147    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:21.294322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:21.287228    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:21.294322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:21.293655    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:21.331161 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:21.331126    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:21.331396 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:21.331182    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:21.415585 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:21.415548    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:22.298393 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:22.298204    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:22.298393 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:22.298267    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:22.330024 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:22.329990    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:22.330226 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:22.330050    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:23.301936 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:23.301894    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:23.302455 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:23.301964    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:23.328608 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:23.328571    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:23.328794 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:23.328631    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:24.288316 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:24.287994    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:24.288316 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:24.288054    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:24.329865 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:24.329826    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:24.330477 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:24.329886    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:25.292322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:25.291563    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 21:59:25.292322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:25.291643    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 21:59:25.328136 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:25.328097    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 21:59:25.328365 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:25.328181    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 21:59:26.916589 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.916550    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:26.917319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.917283    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:26.918183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.918157    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 21:59:26.936181 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.936144    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:26.937188 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.937159    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:26.938412 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:26.938387    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 21:59:30.604549 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:30.604482    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.145791 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.145746    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.280328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.259898    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 21:59:31.335566 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.335523    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.530529 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.335661    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 21:59:31.530813 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.335829    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:59:31.530942 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.335858    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.541451 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.533338    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.542347 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.542290    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 01 21:59:31.566441 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.563985    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:59:31.566980 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.565226    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.622067 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.607521    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:31.712323 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.711797    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:59:31.940383 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:31.937004    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.413322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.255496    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.413322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.270097    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:59:32.413322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.282179    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:32.413322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.315435    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:59:32.550321 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.546276    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:32.551068 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.551030    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:32.589495 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.589449    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.605806 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.589830    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:32.606171 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.589864    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.606345 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.589887    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.638482 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.638424    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:32.743405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.743360    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 21:59:32.899940 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.837090    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:59:33.026405 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.900881    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:59:33.058369 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.030626    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:59:33.061494 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.987788    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.065818 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:32.944765    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.207362 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.207332    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:59:33.207704 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.207674    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:59:33.207908 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.207893    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:59:33.209262 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.209244    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:59:33.261333 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.257407    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.318973 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.282582    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.619413 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.608779    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.798765 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.798738    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 21:59:33.861515 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.861470    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.896834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.896723    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 01 21:59:33.913864 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.901624    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.915365 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.901786    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:33.916518 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.901807    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 21:59:33.917623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:33.917591    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 21:59:34.072918 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.072876    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 21:59:34.112644 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.112610    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 21:59:34.174381 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.174209    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 21:59:34.194765 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.193962    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 21:59:34.292337 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.284944    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 21:59:34.292337 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.286270    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 21:59:34.292337 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.289700    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 21:59:34.308614 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.308582    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 21:59:34.318958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.318872    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 21:59:34.328470 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:34.328441    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 21:59:34.832559 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:34.832502    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 21:59:43.219645 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.219607    5845 generic.go:356] "Generic (PLEG): container finished" podID="57e7747b-833b-4c5b-9edb-871a637e2458" containerID="04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572" exitCode=0
Nov 01 21:59:43.220190 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.219665    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerDied","Data":"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"}
Nov 01 21:59:43.220190 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.220169    5845 scope.go:122] "RemoveContainer" containerID="04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"
Nov 01 21:59:43.221746 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.221717    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:43.221834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.221788    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:43.221834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.221821    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:43.632969 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.632886    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 21:59:43.637600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.637573    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.646584 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.646046    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1aee4437-serving-cert\""
Nov 01 21:59:43.650224 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.650194    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 21:59:43.830318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.828061    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6c5\" (UniqueName: \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.830318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.828165    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.830318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.828457    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.830318 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.828566    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.933198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.932320    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6c5\" (UniqueName: \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.933198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.932398    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.933198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.932636    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.933198 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.932750    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.933958 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.933922    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.949512 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.942124    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.969327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.964047    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6c5\" (UniqueName: \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:43.969327 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:43.964813    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\") pod \"maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:44.235798 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.235761    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerStarted","Data":"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"}
Nov 01 21:59:44.272000 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.271883    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:44.802500 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.802473    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 21:59:44.803497 ip-10-0-135-146 kubenswrapper[5845]: W1101 21:59:44.803466    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9371606b_063c_4de2_a16d_4d75c46ab821.slice/crio-4fccd3ac52aff3ed4c66f86788939533095077ad83355e0d3722e1ef8b7f4945 WatchSource:0}: Error finding container 4fccd3ac52aff3ed4c66f86788939533095077ad83355e0d3722e1ef8b7f4945: Status 404 returned error can't find the container with id 4fccd3ac52aff3ed4c66f86788939533095077ad83355e0d3722e1ef8b7f4945
Nov 01 21:59:44.863324 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.858702    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:44.869504 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.869456    5845 patch_prober.go:29] interesting pod/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 21:59:44.869682 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:44.869536    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 21:59:45.272330 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:45.261016    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" event={"ID":"9371606b-063c-4de2-a16d-4d75c46ab821","Type":"ContainerStarted","Data":"4fccd3ac52aff3ed4c66f86788939533095077ad83355e0d3722e1ef8b7f4945"}
Nov 01 21:59:45.865319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:45.863160    5845 patch_prober.go:29] interesting pod/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 21:59:45.865319 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:45.863217    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 21:59:46.278392 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:46.278343    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" event={"ID":"9371606b-063c-4de2-a16d-4d75c46ab821","Type":"ContainerStarted","Data":"d7195ce52a0aa80533e1ecc2c8f9ca9a5f6f0eb6c54acf4b4e6dd210b98e499f"}
Nov 01 21:59:46.278815 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:46.278519    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:46.300014 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:46.299963    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" podStartSLOduration=2.8275792060000002 podStartE2EDuration="3.299950299s" totalImagesPullingTime="472.371093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:46.298342794 +0000 UTC m=+889.554522558" watchObservedRunningTime="2026-11-01 21:59:46.299950299 +0000 UTC m=+889.556130062"
Nov 01 21:59:46.870326 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:46.870262    5845 patch_prober.go:29] interesting pod/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 21:59:46.870516 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:46.870337    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 21:59:47.859823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:47.858660    5845 patch_prober.go:29] interesting pod/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 21:59:47.859823 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:47.858722    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 21:59:48.865322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:48.863653    5845 patch_prober.go:29] interesting pod/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 01 21:59:48.865322 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:48.863756    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 01 21:59:49.026643 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.026605    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:49.027211 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.027177    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" podUID="cfa10098-ec06-46cf-a7bb-3213eddca65d" containerName="maas-api" containerID="cri-o://ed69c9ccce57c8ff1c579e64af5be1b265317f68dd4a80781f808f217b232b04" gracePeriod=30
Nov 01 21:59:49.182567 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.182533    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:49.182834 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.182776    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" podUID="912476b8-9e30-4f52-9019-3e7daba96f50" containerName="maas-api" containerID="cri-o://430a57a2a13ae96de44087fb18bf32d5d75a352f7d8264ac2de963358e75709d" gracePeriod=30
Nov 01 21:59:49.302893 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.302843    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:49.312431 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.312403    5845 generic.go:356] "Generic (PLEG): container finished" podID="912476b8-9e30-4f52-9019-3e7daba96f50" containerID="430a57a2a13ae96de44087fb18bf32d5d75a352f7d8264ac2de963358e75709d" exitCode=0
Nov 01 21:59:49.312553 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.312424    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" event={"ID":"912476b8-9e30-4f52-9019-3e7daba96f50","Type":"ContainerDied","Data":"430a57a2a13ae96de44087fb18bf32d5d75a352f7d8264ac2de963358e75709d"}
Nov 01 21:59:49.314522 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.314492    5845 generic.go:356] "Generic (PLEG): container finished" podID="cfa10098-ec06-46cf-a7bb-3213eddca65d" containerID="ed69c9ccce57c8ff1c579e64af5be1b265317f68dd4a80781f808f217b232b04" exitCode=0
Nov 01 21:59:49.314620 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.314557    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" event={"ID":"cfa10098-ec06-46cf-a7bb-3213eddca65d","Type":"ContainerDied","Data":"ed69c9ccce57c8ff1c579e64af5be1b265317f68dd4a80781f808f217b232b04"}
Nov 01 21:59:49.632134 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.629130    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:49.677874 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.677813    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") pod \"cfa10098-ec06-46cf-a7bb-3213eddca65d\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") "
Nov 01 21:59:49.685840 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.685736    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls" pod "cfa10098-ec06-46cf-a7bb-3213eddca65d" (UID: "cfa10098-ec06-46cf-a7bb-3213eddca65d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:49.722218 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.721392    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:49.779581 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.779491    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\" (UniqueName: \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\") pod \"cfa10098-ec06-46cf-a7bb-3213eddca65d\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") "
Nov 01 21:59:49.779759 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.779740    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\") pod \"cfa10098-ec06-46cf-a7bb-3213eddca65d\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") "
Nov 01 21:59:49.780000 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.779966    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\") pod \"cfa10098-ec06-46cf-a7bb-3213eddca65d\" (UID: \"cfa10098-ec06-46cf-a7bb-3213eddca65d\") "
Nov 01 21:59:49.780103 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.780012    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\") pod \"912476b8-9e30-4f52-9019-3e7daba96f50\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") "
Nov 01 21:59:49.780252 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.780204    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\" (UniqueName: \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\") pod \"912476b8-9e30-4f52-9019-3e7daba96f50\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") "
Nov 01 21:59:49.780461 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.780404    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca" pod "cfa10098-ec06-46cf-a7bb-3213eddca65d" (UID: "cfa10098-ec06-46cf-a7bb-3213eddca65d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:49.780941 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.780916    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.781035 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.780945    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa10098-ec06-46cf-a7bb-3213eddca65d-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.782069 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.782045    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg" pod "cfa10098-ec06-46cf-a7bb-3213eddca65d" (UID: "cfa10098-ec06-46cf-a7bb-3213eddca65d"). InnerVolumeSpecName "kube-api-access-jnvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:49.782187 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.782157    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs" pod "cfa10098-ec06-46cf-a7bb-3213eddca65d" (UID: "cfa10098-ec06-46cf-a7bb-3213eddca65d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:49.782653 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.782629    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs" pod "912476b8-9e30-4f52-9019-3e7daba96f50" (UID: "912476b8-9e30-4f52-9019-3e7daba96f50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:49.782857 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.782836    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg" pod "912476b8-9e30-4f52-9019-3e7daba96f50" (UID: "912476b8-9e30-4f52-9019-3e7daba96f50"). InnerVolumeSpecName "kube-api-access-r27zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:49.881534 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.881486    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\") pod \"912476b8-9e30-4f52-9019-3e7daba96f50\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") "
Nov 01 21:59:49.882009 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.881834    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\") pod \"912476b8-9e30-4f52-9019-3e7daba96f50\" (UID: \"912476b8-9e30-4f52-9019-3e7daba96f50\") "
Nov 01 21:59:49.882183 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882158    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca" pod "912476b8-9e30-4f52-9019-3e7daba96f50" (UID: "912476b8-9e30-4f52-9019-3e7daba96f50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:49.882694 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882664    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27zg\" (UniqueName: \"kubernetes.io/projected/912476b8-9e30-4f52-9019-3e7daba96f50-kube-api-access-r27zg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.882694 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882696    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/912476b8-9e30-4f52-9019-3e7daba96f50-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.882850 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882712    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvzg\" (UniqueName: \"kubernetes.io/projected/cfa10098-ec06-46cf-a7bb-3213eddca65d-kube-api-access-jnvzg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.882850 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882726    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa10098-ec06-46cf-a7bb-3213eddca65d-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.882850 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.882739    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:49.883692 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.883666    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls" pod "912476b8-9e30-4f52-9019-3e7daba96f50" (UID: "912476b8-9e30-4f52-9019-3e7daba96f50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 21:59:49.975465 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.975428    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:49.984142 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:49.984108    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/912476b8-9e30-4f52-9019-3e7daba96f50-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:50.323041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.322948    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"
Nov 01 21:59:50.323041 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.322979    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5" event={"ID":"912476b8-9e30-4f52-9019-3e7daba96f50","Type":"ContainerDied","Data":"b8b771fc6cd16c2f1b1db399f608b517b6d7df5f2232d52c175438e0723c6342"}
Nov 01 21:59:50.323259 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.323047    5845 scope.go:122] "RemoveContainer" containerID="430a57a2a13ae96de44087fb18bf32d5d75a352f7d8264ac2de963358e75709d"
Nov 01 21:59:50.324588 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.324563    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"
Nov 01 21:59:50.324684 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.324656    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx" event={"ID":"cfa10098-ec06-46cf-a7bb-3213eddca65d","Type":"ContainerDied","Data":"68a77635e6c775692d9134fde38cdea55e541716784c4878e86773031d969daa"}
Nov 01 21:59:50.329526 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.329504    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:50.337251 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.337234    5845 scope.go:122] "RemoveContainer" containerID="ed69c9ccce57c8ff1c579e64af5be1b265317f68dd4a80781f808f217b232b04"
Nov 01 21:59:50.384375 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.384338    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:50.390575 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.389838    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-13364262-777d6b4cd-c8nnx"]
Nov 01 21:59:50.403186 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.403155    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:50.411335 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.408671    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681db477-7c889cc6f4-h9kz5"]
Nov 01 21:59:50.779589 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.779555    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:50.864502 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.864456    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912476b8-9e30-4f52-9019-3e7daba96f50" path="/var/lib/kubelet/pods/912476b8-9e30-4f52-9019-3e7daba96f50/volumes"
Nov 01 21:59:50.865156 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:50.865135    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa10098-ec06-46cf-a7bb-3213eddca65d" path="/var/lib/kubelet/pods/cfa10098-ec06-46cf-a7bb-3213eddca65d/volumes"
Nov 01 21:59:51.334178 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.334145    5845 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" secret="" err="secret \"e2e-ait-13364262-openshift-default-dockercfg-hk9r6\" not found"
Nov 01 21:59:51.424656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.424614    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 21:59:51.425281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425264    5845 cpu_manager.go:414] "containerMap: removing container" podUID="912476b8-9e30-4f52-9019-3e7daba96f50" containerName="maas-api"
Nov 01 21:59:51.425281 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425281    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912476b8-9e30-4f52-9019-3e7daba96f50" containerName="maas-api"
Nov 01 21:59:51.425425 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425351    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cfa10098-ec06-46cf-a7bb-3213eddca65d" containerName="maas-api"
Nov 01 21:59:51.425425 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425359    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa10098-ec06-46cf-a7bb-3213eddca65d" containerName="maas-api"
Nov 01 21:59:51.425498 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425489    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="912476b8-9e30-4f52-9019-3e7daba96f50" containerName="maas-api"
Nov 01 21:59:51.425533 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.425506    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa10098-ec06-46cf-a7bb-3213eddca65d" containerName="maas-api"
Nov 01 21:59:51.430600 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.430577    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.441343 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.441267    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d5713372-openshift-default-dockercfg-vkq8w\""
Nov 01 21:59:51.445064 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.445026    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 21:59:51.525373 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525330    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525587 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525393    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525587 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525465    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525587 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525502    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525587 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525540    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525869 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525616    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525869 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525677    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525869 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525779    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znsdh\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.525869 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.525864    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628513    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628576    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628643    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628720    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628794    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628853    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.628953    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znsdh\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.629388 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.629034    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.633328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.630152    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.633328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.630440    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.633328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.631268    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.633328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.631357    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.633328 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.631600    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.634615 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.634591    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.649333 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.646196    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znsdh\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.666656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.666373    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.666656 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.666594    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\") pod \"e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") " pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:51.745883 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:51.745851    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:52.048941 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.048875    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 21:59:52.058543 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.058505    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:52.058655 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.058597    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:52.058655 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.058639    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 21:59:52.293733 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.293705    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 21:59:52.346087 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.346001    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy" containerID="cri-o://4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083" gracePeriod=30
Nov 01 21:59:52.346970 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.346941    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" event={"ID":"a7c800a0-789a-45b2-9048-ce6989b11ba6","Type":"ContainerStarted","Data":"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"}
Nov 01 21:59:52.347087 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.346988    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" event={"ID":"a7c800a0-789a-45b2-9048-ce6989b11ba6","Type":"ContainerStarted","Data":"4160d345eff6beea8b2d147a9b6397fa7cc88ffdeb9a82451047219522141d45"}
Nov 01 21:59:52.400051 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.399999    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" podStartSLOduration=1.39998452 podStartE2EDuration="1.39998452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 21:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:59:52.373907188 +0000 UTC m=+895.630086951" watchObservedRunningTime="2026-11-01 21:59:52.39998452 +0000 UTC m=+895.656164283"
Nov 01 21:59:52.747444 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:52.747393    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:53.752276 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:53.752246    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:54.359891 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:54.359862    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:54.360904 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:54.360882    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 21:59:56.992623 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:56.992597    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:59:56.993097 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:56.992597    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 21:59:57.661815 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.661788    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:57.698357 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698321    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698528 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698510    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698575 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698561    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698691 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698674    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698732 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698716    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698801 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698788    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698840 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698824    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698889 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698875    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.698998 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.698982    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\") pod \"57e7747b-833b-4c5b-9edb-871a637e2458\" (UID: \"57e7747b-833b-4c5b-9edb-871a637e2458\") "
Nov 01 21:59:57.700396 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.700284    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:59:57.700680 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.700609    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:59:57.700800 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.700718    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 21:59:57.700889 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.700793    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:59:57.701077 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.701050    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:59:57.702657 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.702629    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:57.702991 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.702969    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 21:59:57.703226 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.703202    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 21:59:57.704384 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.704357    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk" pod "57e7747b-833b-4c5b-9edb-871a637e2458" (UID: "57e7747b-833b-4c5b-9edb-871a637e2458"). InnerVolumeSpecName "kube-api-access-9rrzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 21:59:57.800923 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800888    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.800923 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800920    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.800923 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800931    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800941    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrzk\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-kube-api-access-9rrzk\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800951    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800960    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57e7747b-833b-4c5b-9edb-871a637e2458-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800968    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57e7747b-833b-4c5b-9edb-871a637e2458-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800977    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57e7747b-833b-4c5b-9edb-871a637e2458-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:57.801167 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:57.800989    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57e7747b-833b-4c5b-9edb-871a637e2458-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 21:59:58.392842 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.392800    5845 generic.go:356] "Generic (PLEG): container finished" podID="57e7747b-833b-4c5b-9edb-871a637e2458" containerID="4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083" exitCode=0
Nov 01 21:59:58.393361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.392859    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"
Nov 01 21:59:58.393361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.392885    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerDied","Data":"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"}
Nov 01 21:59:58.393361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.392960    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf" event={"ID":"57e7747b-833b-4c5b-9edb-871a637e2458","Type":"ContainerDied","Data":"140b38adfc18c9b9547ffa3e1ff57e9cbdf5595b55dc22e969c9ce6dabf687a0"}
Nov 01 21:59:58.393361 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.392998    5845 scope.go:122] "RemoveContainer" containerID="4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"
Nov 01 21:59:58.403611 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.403590    5845 scope.go:122] "RemoveContainer" containerID="04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"
Nov 01 21:59:58.413218 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.413200    5845 scope.go:122] "RemoveContainer" containerID="4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"
Nov 01 21:59:58.413854 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:58.413618    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083\": container with ID starting with 4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083 not found: ID does not exist" containerID="4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"
Nov 01 21:59:58.413854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.413724    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083"} err="failed to get container status \"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083\": rpc error: code = NotFound desc = could not find container \"4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083\": container with ID starting with 4edc9048c8177a7828f1736d127fcd1ed20e89cfeba0bd4d2776c43a41c05083 not found: ID does not exist"
Nov 01 21:59:58.413854 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.413770    5845 scope.go:122] "RemoveContainer" containerID="04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"
Nov 01 21:59:58.414731 ip-10-0-135-146 kubenswrapper[5845]: E1101 21:59:58.414697    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572\": container with ID starting with 04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572 not found: ID does not exist" containerID="04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"
Nov 01 21:59:58.414864 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.414738    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572"} err="failed to get container status \"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572\": rpc error: code = NotFound desc = could not find container \"04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572\": container with ID starting with 04117bf10fc633a9315992761f79ef40a4babb89ca44087d4da897d13c88c572 not found: ID does not exist"
Nov 01 21:59:58.418868 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.418811    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:58.422415 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.422391    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-13364262-openshift-default-84754cd9dc-tnjlf"]
Nov 01 21:59:58.864457 ip-10-0-135-146 kubenswrapper[5845]: I1101 21:59:58.864422    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" path="/var/lib/kubelet/pods/57e7747b-833b-4c5b-9edb-871a637e2458/volumes"
Nov 01 22:00:00.188656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.187943    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:00.189139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.188998    5845 cpu_manager.go:414] "containerMap: removing container" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.189139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.189019    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.189139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.189034    5845 cpu_manager.go:414] "containerMap: removing container" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.189139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.189043    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.190317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.189950    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.190317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.189975    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e7747b-833b-4c5b-9edb-871a637e2458" containerName="istio-proxy"
Nov 01 22:00:00.203350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.196450    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.203350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.201257    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:00.243773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.243725    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdn6\" (UniqueName: \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.243946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.243782    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.273027 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.272980    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\""
Nov 01 22:00:00.347970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.346617    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdn6\" (UniqueName: \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.347970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.346674    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.347970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.347704    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.359931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.359905    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdn6\" (UniqueName: \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\") pod \"maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t\" (UID: \"21492635-d929-458c-b196-854ba030153f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.516354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.516286    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:00.798310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.798263    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:00.799532 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:00.799502    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21492635_d929_458c_b196_854ba030153f.slice/crio-cc809b07b47cdc39116239ec5ec54c324ac4a795875a911654330116bef4212e WatchSource:0}: Error finding container cc809b07b47cdc39116239ec5ec54c324ac4a795875a911654330116bef4212e: Status 404 returned error can't find the container with id cc809b07b47cdc39116239ec5ec54c324ac4a795875a911654330116bef4212e
Nov 01 22:00:00.927769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.927731    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:00.935229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.933678    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:00.938060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.938031    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d5713372-serving-cert\""
Nov 01 22:00:00.938687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:00.938659    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:01.056360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.056269    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.056360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.056341    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.056533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.056454    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrg6\" (UniqueName: \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.056533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.056491    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.158346 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.158285    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.158538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.158370    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.158538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.158469    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrg6\" (UniqueName: \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.158538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.158505    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.159104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.159081    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.161614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.161587    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.161725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.161663    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.171285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.171264    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrg6\" (UniqueName: \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\") pod \"maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.254493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.254431    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:01.430373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.430287    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerStarted","Data":"cc809b07b47cdc39116239ec5ec54c324ac4a795875a911654330116bef4212e"}
Nov 01 22:00:01.572284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:01.572248    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:02.438031 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:02.437993    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" event={"ID":"e93b35c8-c836-4f51-8234-c466d02754ef","Type":"ContainerStarted","Data":"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"}
Nov 01 22:00:02.438445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:02.438046    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" event={"ID":"e93b35c8-c836-4f51-8234-c466d02754ef","Type":"ContainerStarted","Data":"5b85c34eae4a19d50f1f4df0101828651b08ead18a918f87c5ebb3b0f827d446"}
Nov 01 22:00:02.438445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:02.438075    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:02.518810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:02.518709    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" podStartSLOduration=1.873495017 podStartE2EDuration="2.518691464s" totalImagesPullingTime="645.196447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:02.515213489 +0000 UTC m=+905.771393250" watchObservedRunningTime="2026-11-01 22:00:02.518691464 +0000 UTC m=+905.774871229"
Nov 01 22:00:03.087159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.087125    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:00:03.093062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.093031    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.108967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.108938    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:00:03.199834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.199781    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.199940    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200106 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200077    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200180    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200234    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljzm\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200265    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200364    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200512    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.200575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.200543    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.301812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.301779    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.301840    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.301866    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljzm\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.301885    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.301919    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302021    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302039    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302124    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302189    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302219    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302239    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302700    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.302958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.302909    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.303145 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.303116    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.304891 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.304867    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.305238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.305220    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.314732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.314708    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.315275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.315246    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljzm\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\") pod \"e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") " pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.451099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.450964    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:03.747611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.747579    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:00:03.750502 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:03.750462    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0388a06b_168c_42e5_93b6_8b2776675272.slice/crio-fdb86a281b606fa5742e197d434db73834fa06b503ea2e6ef08d22494f71c0ca WatchSource:0}: Error finding container fdb86a281b606fa5742e197d434db73834fa06b503ea2e6ef08d22494f71c0ca: Status 404 returned error can't find the container with id fdb86a281b606fa5742e197d434db73834fa06b503ea2e6ef08d22494f71c0ca
Nov 01 22:00:03.754022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.753988    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:03.754137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.754068    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:03.754137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:03.754114    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:04.457137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:04.457104    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" event={"ID":"0388a06b-168c-42e5-93b6-8b2776675272","Type":"ContainerStarted","Data":"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"}
Nov 01 22:00:04.457668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:04.457161    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" event={"ID":"0388a06b-168c-42e5-93b6-8b2776675272","Type":"ContainerStarted","Data":"fdb86a281b606fa5742e197d434db73834fa06b503ea2e6ef08d22494f71c0ca"}
Nov 01 22:00:04.487897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:04.487843    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podStartSLOduration=1.487827931 podStartE2EDuration="1.487827931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:04.485818031 +0000 UTC m=+907.741997794" watchObservedRunningTime="2026-11-01 22:00:04.487827931 +0000 UTC m=+907.744007695"
Nov 01 22:00:04.824901 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:04.824866    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:00:05.468322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:05.454915    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:05.468322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:05.461418    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:05.468322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:05.461473    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:06.457319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.456533    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:06.457319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.456594    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:06.605329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.604491    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerStarted","Data":"d66ad100d2b3048dde145e2223409ab7655f1bba51f9698907943a757cbebd65"}
Nov 01 22:00:06.731316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.731016    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" podStartSLOduration=2.220674453 podStartE2EDuration="6.730980498s" totalImagesPullingTime="4.510306045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:06.654693613 +0000 UTC m=+909.910873377" watchObservedRunningTime="2026-11-01 22:00:06.730980498 +0000 UTC m=+909.987160262"
Nov 01 22:00:06.945449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.945398    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:00:07.041434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:06.950609    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 22:00:07.168323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.163861    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.278537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.278479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.310381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.306159    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.310381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.306493    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.310381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.307194    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.461320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.457740    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:07.461320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.457802    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:07.665415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.664320    5845 generic.go:356] "Generic (PLEG): container finished" podID="21492635-d929-458c-b196-854ba030153f" containerID="d66ad100d2b3048dde145e2223409ab7655f1bba51f9698907943a757cbebd65" exitCode=60
Nov 01 22:00:07.665415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.664451    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerDied","Data":"d66ad100d2b3048dde145e2223409ab7655f1bba51f9698907943a757cbebd65"}
Nov 01 22:00:07.665415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.664953    5845 scope.go:122] "RemoveContainer" containerID="d66ad100d2b3048dde145e2223409ab7655f1bba51f9698907943a757cbebd65"
Nov 01 22:00:07.751646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.749422    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.818569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.816381    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:00:07.922515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.922390    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.963455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.963416    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.963772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.962793    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.966256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.964106    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:00:07.966256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.962753    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:07.966256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:07.964985    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:00:08.173703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.173599    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.284378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.281961    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.284378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.282755    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:00:08.311460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.311414    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.362365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.362335    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:00:08.388364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.316405    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.389062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.389033    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:00:08.554330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.548139    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:08.554330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.548199    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:08.652395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.633469    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:08.775008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.774280    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.775008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.774623    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:00:08.828627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.827572    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.828627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.827998    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:00:08.971811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.971568    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.980383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.972133    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:00:08.983395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.972166    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:08.991051 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:08.972195    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:00:09.119729 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.072874    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:00:09.120036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.120015    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:00:09.185699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.185643    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:09.186699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.186372    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:00:09.383700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.383587    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:09.393320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.385940    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:00:09.605273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.559844    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:00:09.668955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.651580    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:09.668955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.651662    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:09.746784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.743121    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 01 22:00:09.784649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.783531    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:00:09.835502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.835443    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.056385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.977595    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:10.056385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.978691    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 01 22:00:10.056385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:09.988770    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.056385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.009836    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.153508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.153464    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.187816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.187673    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.543742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.516548    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:10.543742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.516620    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:10.543742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.519081    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.572326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.547037    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.602579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.591485    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:10.672368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.668680    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:00:10.672368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.668741    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:10.739386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.704179    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 01 22:00:10.804508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.804447    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:10.895341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.893681    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:00:10.980529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:10.980482    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.055384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.015968    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.113462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.056214    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:11.188324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.185754    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:11.211499 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.211456    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:11.634324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.610837    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.748355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.748279    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.752445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.752403    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.835288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.835230    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:11.888374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.848924    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:11.888374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.880763    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:11.888374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.868026    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:00:11.931451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.912210    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 01 22:00:11.980641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.937134    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:11.990451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:11.990408    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.124292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.124240    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:00:12.137766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.128896    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.203737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.203625    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:00:12.244830 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.214212    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.404068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.404007    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.442099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.442064    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:00:12.455385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.451417    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:12.455385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.451479    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:12.478273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.478234    5845 generic.go:356] "Generic (PLEG): container finished" podID="21492635-d929-458c-b196-854ba030153f" containerID="23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f" exitCode=60
Nov 01 22:00:12.478570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.478549    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerDied","Data":"23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f"}
Nov 01 22:00:12.478720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.478706    5845 scope.go:122] "RemoveContainer" containerID="d66ad100d2b3048dde145e2223409ab7655f1bba51f9698907943a757cbebd65"
Nov 01 22:00:12.479510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.479490    5845 scope.go:122] "RemoveContainer" containerID="23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f"
Nov 01 22:00:12.480115 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:12.480088    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t_odh-ai-gateway-infra(21492635-d929-458c-b196-854ba030153f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" podUID="21492635-d929-458c-b196-854ba030153f"
Nov 01 22:00:12.669328 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.668940    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.752562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.752517    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:12.768572 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.753880    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:00:12.886327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:12.884104    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:00:13.069610 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.069482    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.071663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.071618    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 22:00:13.106337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.030519    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": context deadline exceeded"
Nov 01 22:00:13.201322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.181657    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.219409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.219359    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.298937 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.298889    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.451733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.451635    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:13.535568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.472048    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:13.552231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.552202    5845 scope.go:122] "RemoveContainer" containerID="23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f"
Nov 01 22:00:13.559196 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:13.559158    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t_odh-ai-gateway-infra(21492635-d929-458c-b196-854ba030153f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" podUID="21492635-d929-458c-b196-854ba030153f"
Nov 01 22:00:13.718402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.701730    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:00:13.718402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.703954    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.718402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.711912    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:00:13.718402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.714958    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:00:13.774457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.774411    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.839958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.839930    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:00:13.870956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.870596    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:00:13.897663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.897620    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:00:13.907531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.901000    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:00:13.995386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:13.991386    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:00:14.041551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:14.013806    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:00:14.041551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:14.037272    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:00:14.452188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:14.452081    5845 patch_prober.go:29] interesting pod/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 22:00:14.452188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:14.452143    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 22:00:15.437020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.436970    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 22:00:15.437933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.437883    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" podUID="9371606b-063c-4de2-a16d-4d75c46ab821" containerName="maas-api" containerID="cri-o://d7195ce52a0aa80533e1ecc2c8f9ca9a5f6f0eb6c54acf4b4e6dd210b98e499f" gracePeriod=30
Nov 01 22:00:15.675662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.675628    5845 generic.go:356] "Generic (PLEG): container finished" podID="9371606b-063c-4de2-a16d-4d75c46ab821" containerID="d7195ce52a0aa80533e1ecc2c8f9ca9a5f6f0eb6c54acf4b4e6dd210b98e499f" exitCode=0
Nov 01 22:00:15.675812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.675792    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" event={"ID":"9371606b-063c-4de2-a16d-4d75c46ab821","Type":"ContainerDied","Data":"d7195ce52a0aa80533e1ecc2c8f9ca9a5f6f0eb6c54acf4b4e6dd210b98e499f"}
Nov 01 22:00:15.830353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.826693    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:15.840153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.838613    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:15.859442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.859367    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:00:15.932418 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.932389    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 22:00:15.960104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.960070    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\" (UniqueName: \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\") pod \"9371606b-063c-4de2-a16d-4d75c46ab821\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") "
Nov 01 22:00:15.960516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.960493    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\") pod \"9371606b-063c-4de2-a16d-4d75c46ab821\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") "
Nov 01 22:00:15.961423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.960699    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\") pod \"9371606b-063c-4de2-a16d-4d75c46ab821\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") "
Nov 01 22:00:15.961423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.961107    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\") pod \"9371606b-063c-4de2-a16d-4d75c46ab821\" (UID: \"9371606b-063c-4de2-a16d-4d75c46ab821\") "
Nov 01 22:00:15.963322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.963286    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca" pod "9371606b-063c-4de2-a16d-4d75c46ab821" (UID: "9371606b-063c-4de2-a16d-4d75c46ab821"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:00:15.965425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.965403    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs" pod "9371606b-063c-4de2-a16d-4d75c46ab821" (UID: "9371606b-063c-4de2-a16d-4d75c46ab821"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:00:15.967760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.967724    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5" pod "9371606b-063c-4de2-a16d-4d75c46ab821" (UID: "9371606b-063c-4de2-a16d-4d75c46ab821"). InnerVolumeSpecName "kube-api-access-hn6c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:15.973187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:15.973163    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls" pod "9371606b-063c-4de2-a16d-4d75c46ab821" (UID: "9371606b-063c-4de2-a16d-4d75c46ab821"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:00:16.063927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.063826    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:16.063927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.063864    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9371606b-063c-4de2-a16d-4d75c46ab821-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:16.063927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.063879    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9371606b-063c-4de2-a16d-4d75c46ab821-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:16.063927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.063893    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn6c5\" (UniqueName: \"kubernetes.io/projected/9371606b-063c-4de2-a16d-4d75c46ab821-kube-api-access-hn6c5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:16.732264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.732231    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"
Nov 01 22:00:16.743746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.743708    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw" event={"ID":"9371606b-063c-4de2-a16d-4d75c46ab821","Type":"ContainerDied","Data":"4fccd3ac52aff3ed4c66f86788939533095077ad83355e0d3722e1ef8b7f4945"}
Nov 01 22:00:16.744001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.743985    5845 scope.go:122] "RemoveContainer" containerID="d7195ce52a0aa80533e1ecc2c8f9ca9a5f6f0eb6c54acf4b4e6dd210b98e499f"
Nov 01 22:00:16.813376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.813342    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 22:00:16.820026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.819972    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1aee4437-55694d97d4-d4grw"]
Nov 01 22:00:16.869952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:16.869910    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9371606b-063c-4de2-a16d-4d75c46ab821" path="/var/lib/kubelet/pods/9371606b-063c-4de2-a16d-4d75c46ab821/volumes"
Nov 01 22:00:24.860967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:24.860931    5845 scope.go:122] "RemoveContainer" containerID="23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f"
Nov 01 22:00:25.169252 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.169183    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:00:25.169896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.169880    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9371606b-063c-4de2-a16d-4d75c46ab821" containerName="maas-api"
Nov 01 22:00:25.169962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.169898    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9371606b-063c-4de2-a16d-4d75c46ab821" containerName="maas-api"
Nov 01 22:00:25.170009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.170000    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9371606b-063c-4de2-a16d-4d75c46ab821" containerName="maas-api"
Nov 01 22:00:25.202100 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.202070    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:00:25.202269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.202166    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.206168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.206134    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1783e3-openshift-default-dockercfg-kwmww\""
Nov 01 22:00:25.268070 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268044    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kjw\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268086    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268187    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268328    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268356    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268503    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268551    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268689    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.268874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.268783    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389363    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kjw\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389430    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389498    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389594    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389618    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389727    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389767    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389868    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.390410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.389941    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.394273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.393855    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.394647 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.394625    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.394979 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.394960    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.395638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.395617    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.395962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.395944    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.428451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.424105    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.440987 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.440957    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kjw\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.454210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.451811    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.454210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.452078    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.518633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.518563    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:25.826103 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:25.826071    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426bc111_0f0a_4c63_a50a_519e47476539.slice/crio-3709df0c16b83d3f2f7063a970977a97de355cbe4d81a05bd808fd813ef50c51 WatchSource:0}: Error finding container 3709df0c16b83d3f2f7063a970977a97de355cbe4d81a05bd808fd813ef50c51: Status 404 returned error can't find the container with id 3709df0c16b83d3f2f7063a970977a97de355cbe4d81a05bd808fd813ef50c51
Nov 01 22:00:25.829234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829089    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:00:25.829355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829243    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:25.829428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829360    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:25.829428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829396    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:25.829545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829515    5845 generic.go:356] "Generic (PLEG): container finished" podID="21492635-d929-458c-b196-854ba030153f" containerID="dad18bd7b968d1eef42f2835bc3fc44a1399093ca8fd659b1c4ce28eaaf4028b" exitCode=60
Nov 01 22:00:25.829605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829580    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerDied","Data":"dad18bd7b968d1eef42f2835bc3fc44a1399093ca8fd659b1c4ce28eaaf4028b"}
Nov 01 22:00:25.829652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.829641    5845 scope.go:122] "RemoveContainer" containerID="23a395290c32f7f5a44aa43391d501e5b5b62f6a73bcd4a5d37626e24988ab9f"
Nov 01 22:00:25.831161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:25.830142    5845 scope.go:122] "RemoveContainer" containerID="dad18bd7b968d1eef42f2835bc3fc44a1399093ca8fd659b1c4ce28eaaf4028b"
Nov 01 22:00:25.831161 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:25.830504    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t_odh-ai-gateway-infra(21492635-d929-458c-b196-854ba030153f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" podUID="21492635-d929-458c-b196-854ba030153f"
Nov 01 22:00:26.839590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:26.839553    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerStarted","Data":"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"}
Nov 01 22:00:26.840049 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:26.839609    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerStarted","Data":"3709df0c16b83d3f2f7063a970977a97de355cbe4d81a05bd808fd813ef50c51"}
Nov 01 22:00:26.863061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:26.863035    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:26.870688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:26.870640    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podStartSLOduration=1.870623545 podStartE2EDuration="1.870623545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:26.864851537 +0000 UTC m=+930.121031506" watchObservedRunningTime="2026-11-01 22:00:26.870623545 +0000 UTC m=+930.126803309"
Nov 01 22:00:27.106439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.106368    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:27.249756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.249727    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\") pod \"21492635-d929-458c-b196-854ba030153f\" (UID: \"21492635-d929-458c-b196-854ba030153f\") "
Nov 01 22:00:27.249917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.249844    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\" (UniqueName: \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\") pod \"21492635-d929-458c-b196-854ba030153f\" (UID: \"21492635-d929-458c-b196-854ba030153f\") "
Nov 01 22:00:27.250085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.250061    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca" pod "21492635-d929-458c-b196-854ba030153f" (UID: "21492635-d929-458c-b196-854ba030153f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:00:27.250557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.250536    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21492635-d929-458c-b196-854ba030153f-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:27.252078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.252058    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6" pod "21492635-d929-458c-b196-854ba030153f" (UID: "21492635-d929-458c-b196-854ba030153f"). InnerVolumeSpecName "kube-api-access-jkdn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:27.352338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.352293    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdn6\" (UniqueName: \"kubernetes.io/projected/21492635-d929-458c-b196-854ba030153f-kube-api-access-jkdn6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:27.519506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.519477    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:27.525214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.525188    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:27.863123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.863011    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"
Nov 01 22:00:27.863567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.863364    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t" event={"ID":"21492635-d929-458c-b196-854ba030153f","Type":"ContainerDied","Data":"cc809b07b47cdc39116239ec5ec54c324ac4a795875a911654330116bef4212e"}
Nov 01 22:00:27.863567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.863433    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:27.863567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.863467    5845 scope.go:122] "RemoveContainer" containerID="dad18bd7b968d1eef42f2835bc3fc44a1399093ca8fd659b1c4ce28eaaf4028b"
Nov 01 22:00:27.864995 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.864849    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:00:27.916814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.916762    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:27.922028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:27.921998    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b0b7a3-29892840-q6m5t"]
Nov 01 22:00:28.862591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:28.862554    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21492635-d929-458c-b196-854ba030153f" path="/var/lib/kubelet/pods/21492635-d929-458c-b196-854ba030153f/volumes"
Nov 01 22:00:30.570665 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.570632    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:00:30.571341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571324    5845 cpu_manager.go:414] "containerMap: removing container" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.571341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571341    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.571427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571364    5845 cpu_manager.go:414] "containerMap: removing container" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.571427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571369    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.571498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571488    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.571585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.571507    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:30.576590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.576565    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.584672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.584037    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-1783e3\""
Nov 01 22:00:30.586765 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.586740    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:00:30.622656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.622622    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.622831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.622810    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq667\" (UniqueName: \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.743305 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.743249    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq667\" (UniqueName: \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.767866 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.767821    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.769012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.768904    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.811068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.806630    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq667\" (UniqueName: \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\") pod \"payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:30.895446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:30.895363    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:31.182864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:31.182836    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:00:31.183725 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:31.183701    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b6f384_447a_44d8_b2a0_dc547696f39b.slice/crio-554e2fabb6f6e2d966e6aa379f5b520b0fc80a00e100ddf6fa407aeea5be2a08 WatchSource:0}: Error finding container 554e2fabb6f6e2d966e6aa379f5b520b0fc80a00e100ddf6fa407aeea5be2a08: Status 404 returned error can't find the container with id 554e2fabb6f6e2d966e6aa379f5b520b0fc80a00e100ddf6fa407aeea5be2a08
Nov 01 22:00:31.900643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:31.900605    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" event={"ID":"62b6f384-447a-44d8-b2a0-dc547696f39b","Type":"ContainerStarted","Data":"554e2fabb6f6e2d966e6aa379f5b520b0fc80a00e100ddf6fa407aeea5be2a08"}
Nov 01 22:00:31.946485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:31.942988    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:31.946485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:31.943216    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" podUID="e93b35c8-c836-4f51-8234-c466d02754ef" containerName="maas-api" containerID="cri-o://f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f" gracePeriod=30
Nov 01 22:00:32.306320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.305830    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 22:00:32.306320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.306049    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" podUID="a7c800a0-789a-45b2-9048-ce6989b11ba6" containerName="istio-proxy" containerID="cri-o://bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9" gracePeriod=30
Nov 01 22:00:32.502146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.501818    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:32.611993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.611865    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\") pod \"e93b35c8-c836-4f51-8234-c466d02754ef\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") "
Nov 01 22:00:32.612154 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.612081    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\") pod \"e93b35c8-c836-4f51-8234-c466d02754ef\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") "
Nov 01 22:00:32.612217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.612197    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\" (UniqueName: \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\") pod \"e93b35c8-c836-4f51-8234-c466d02754ef\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") "
Nov 01 22:00:32.612522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.612385    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\") pod \"e93b35c8-c836-4f51-8234-c466d02754ef\" (UID: \"e93b35c8-c836-4f51-8234-c466d02754ef\") "
Nov 01 22:00:32.615762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.615736    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca" pod "e93b35c8-c836-4f51-8234-c466d02754ef" (UID: "e93b35c8-c836-4f51-8234-c466d02754ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:00:32.622565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.621095    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93b35c8-c836-4f51-8234-c466d02754ef-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:32.624204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.624144    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs" pod "e93b35c8-c836-4f51-8234-c466d02754ef" (UID: "e93b35c8-c836-4f51-8234-c466d02754ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:00:32.624363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.624256    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls" pod "e93b35c8-c836-4f51-8234-c466d02754ef" (UID: "e93b35c8-c836-4f51-8234-c466d02754ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:00:32.624462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.624378    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6" pod "e93b35c8-c836-4f51-8234-c466d02754ef" (UID: "e93b35c8-c836-4f51-8234-c466d02754ef"). InnerVolumeSpecName "kube-api-access-nzrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:32.723203 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.723122    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:32.723203 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.723161    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrg6\" (UniqueName: \"kubernetes.io/projected/e93b35c8-c836-4f51-8234-c466d02754ef-kube-api-access-nzrg6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:32.723203 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.723177    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93b35c8-c836-4f51-8234-c466d02754ef-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:32.908986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.908948    5845 generic.go:356] "Generic (PLEG): container finished" podID="e93b35c8-c836-4f51-8234-c466d02754ef" containerID="f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f" exitCode=0
Nov 01 22:00:32.909525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.909051    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" event={"ID":"e93b35c8-c836-4f51-8234-c466d02754ef","Type":"ContainerDied","Data":"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"}
Nov 01 22:00:32.909525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.909106    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm" event={"ID":"e93b35c8-c836-4f51-8234-c466d02754ef","Type":"ContainerDied","Data":"5b85c34eae4a19d50f1f4df0101828651b08ead18a918f87c5ebb3b0f827d446"}
Nov 01 22:00:32.909525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.909135    5845 scope.go:122] "RemoveContainer" containerID="f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"
Nov 01 22:00:32.909525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.909283    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"
Nov 01 22:00:32.920143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.920117    5845 scope.go:122] "RemoveContainer" containerID="f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"
Nov 01 22:00:32.920463 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:32.920433    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f\": container with ID starting with f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f not found: ID does not exist" containerID="f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"
Nov 01 22:00:32.920565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.920470    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f"} err="failed to get container status \"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f\": rpc error: code = NotFound desc = could not find container \"f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f\": container with ID starting with f02bc7b537dd00b3b637c492bfdaa579d7eaf979efe453047cf1bc692e574f5f not found: ID does not exist"
Nov 01 22:00:32.930722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.930702    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:32.936228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:32.936207    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d5713372-5ff7dd7f9b-6rgpm"]
Nov 01 22:00:33.505602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.505567    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:00:33.506452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506425    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e93b35c8-c836-4f51-8234-c466d02754ef" containerName="maas-api"
Nov 01 22:00:33.506452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506447    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93b35c8-c836-4f51-8234-c466d02754ef" containerName="maas-api"
Nov 01 22:00:33.506674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506484    5845 cpu_manager.go:414] "containerMap: removing container" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:33.506674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506491    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:33.506674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506667    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93b35c8-c836-4f51-8234-c466d02754ef" containerName="maas-api"
Nov 01 22:00:33.506853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.506695    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="21492635-d929-458c-b196-854ba030153f" containerName="cleanup"
Nov 01 22:00:33.518500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.515887    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.529875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.529849    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-f758d9-openshift-default-dockercfg-w28dk\""
Nov 01 22:00:33.535543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.535509    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:00:33.637163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637130    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637332    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xhnt\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637372    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637439    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637470    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637594 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637560    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637731 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637630    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637731 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637699    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.637850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.637747    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739618    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739787    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xhnt\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739824    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739862    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739894    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739958    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.739995    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.740042    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.740107    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.740805    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.742395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.741796    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.759721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.743406    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.759721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.743909    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.759721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.758913    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.823354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.823262    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.849017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.848951    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.877901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.877837    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:33.910898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:33.907050    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xhnt\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\") pod \"e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") " pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:34.144358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.144255    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:34.456537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.456143    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:00:34.472272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.472234    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:34.472402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.472332    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:34.472402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.472377    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:00:34.820428 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:34.820390    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:00:34.863172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.863137    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93b35c8-c836-4f51-8234-c466d02754ef" path="/var/lib/kubelet/pods/e93b35c8-c836-4f51-8234-c466d02754ef/volumes"
Nov 01 22:00:34.939794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.939759    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" event={"ID":"62b6f384-447a-44d8-b2a0-dc547696f39b","Type":"ContainerStarted","Data":"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"}
Nov 01 22:00:34.940202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.939842    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:34.941456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.941436    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerStarted","Data":"f484bfa3b61df0e2f9dd8eeac4379b86b596c53045c86708874d4c752c1d4564"}
Nov 01 22:00:34.941514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.941472    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerStarted","Data":"e9042e8634157e4b01857026437fe0f552d293823df81ef7bd71ccd85265c52a"}
Nov 01 22:00:34.959509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.959459    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podStartSLOduration=1.573087393 podStartE2EDuration="4.95944674s" totalImagesPullingTime="3.386359347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:34.957687612 +0000 UTC m=+938.213867377" watchObservedRunningTime="2026-11-01 22:00:34.95944674 +0000 UTC m=+938.215626503"
Nov 01 22:00:34.999500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:34.999455    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podStartSLOduration=1.9994426 podStartE2EDuration="1.9994426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:34.994349992 +0000 UTC m=+938.250529756" watchObservedRunningTime="2026-11-01 22:00:34.9994426 +0000 UTC m=+938.255622362"
Nov 01 22:00:35.145258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.145168    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:35.156169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.156138    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:35.695373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.695337    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:00:35.702061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.701165    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.707775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.707746    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-1783e3-serving-cert\""
Nov 01 22:00:35.711122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.710802    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:00:35.783305 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.783266    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.783462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.783401    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79cb\" (UniqueName: \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.783526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.783498    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.783714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.783697    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.888628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.888594    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.888945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.888906    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.889085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.889058    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.889628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.889178    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79cb\" (UniqueName: \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.890001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.889944    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.891610 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.891581    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.892846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.892823    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.901591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.901568    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79cb\" (UniqueName: \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\") pod \"maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:35.952159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.951947    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:35.953057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:35.953033    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:00:36.019514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:36.019479    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:36.323037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:36.323010    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:00:36.324203 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:36.324166    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9240b129_f08e_44df_9f42_da8c7cae75ef.slice/crio-93ba0a615562f180aeb3f648ea6252c91a69522a76a44baea36504736bb3c510 WatchSource:0}: Error finding container 93ba0a615562f180aeb3f648ea6252c91a69522a76a44baea36504736bb3c510: Status 404 returned error can't find the container with id 93ba0a615562f180aeb3f648ea6252c91a69522a76a44baea36504736bb3c510
Nov 01 22:00:36.959600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:36.959568    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" event={"ID":"9240b129-f08e-44df-9f42-da8c7cae75ef","Type":"ContainerStarted","Data":"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"}
Nov 01 22:00:36.960002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:36.959625    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" event={"ID":"9240b129-f08e-44df-9f42-da8c7cae75ef","Type":"ContainerStarted","Data":"93ba0a615562f180aeb3f648ea6252c91a69522a76a44baea36504736bb3c510"}
Nov 01 22:00:36.982927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:36.982872    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" podStartSLOduration=1.500214451 podStartE2EDuration="1.98285981s" totalImagesPullingTime="482.645359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:36.979539714 +0000 UTC m=+940.235719479" watchObservedRunningTime="2026-11-01 22:00:36.98285981 +0000 UTC m=+940.239039572"
Nov 01 22:00:37.607115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.607087    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 22:00:37.639960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.639904    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.639960    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640027    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640066    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640119    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640151    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640196    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:37.640445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640267    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:37.640445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640324    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640374    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640454    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:37.640607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640539    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\") pod \"a7c800a0-789a-45b2-9048-ce6989b11ba6\" (UID: \"a7c800a0-789a-45b2-9048-ce6989b11ba6\") "
Nov 01 22:00:37.640700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640539    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:37.640700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.640566    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:00:37.641274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.641248    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.641402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.641277    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.641402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.641335    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c800a0-789a-45b2-9048-ce6989b11ba6-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.641402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.641352    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.641402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.641369    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.642806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.642779    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "kube-api-access-znsdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:37.642918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.642822    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:00:37.642977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.642915    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:00:37.642977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.642948    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token" pod "a7c800a0-789a-45b2-9048-ce6989b11ba6" (UID: "a7c800a0-789a-45b2-9048-ce6989b11ba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:00:37.742793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.742754    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.742793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.742791    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znsdh\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-kube-api-access-znsdh\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.742793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.742802    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.742999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.742812    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c800a0-789a-45b2-9048-ce6989b11ba6-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:00:37.969288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.969251    5845 generic.go:356] "Generic (PLEG): container finished" podID="a7c800a0-789a-45b2-9048-ce6989b11ba6" containerID="bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9" exitCode=0
Nov 01 22:00:37.969841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.969324    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"
Nov 01 22:00:37.969841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.969330    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" event={"ID":"a7c800a0-789a-45b2-9048-ce6989b11ba6","Type":"ContainerDied","Data":"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"}
Nov 01 22:00:37.969841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.969404    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh" event={"ID":"a7c800a0-789a-45b2-9048-ce6989b11ba6","Type":"ContainerDied","Data":"4160d345eff6beea8b2d147a9b6397fa7cc88ffdeb9a82451047219522141d45"}
Nov 01 22:00:37.969841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.969435    5845 scope.go:122] "RemoveContainer" containerID="bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"
Nov 01 22:00:37.977877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.970225    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:37.993475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.993021    5845 scope.go:122] "RemoveContainer" containerID="bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"
Nov 01 22:00:37.994056 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:00:37.994035    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9\": container with ID starting with bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9 not found: ID does not exist" containerID="bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"
Nov 01 22:00:37.994136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:37.994068    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9"} err="failed to get container status \"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9\": rpc error: code = NotFound desc = could not find container \"bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9\": container with ID starting with bbcff7d104855f7fcdbd11c5590513821c10afb42254e09aaaee11b0d93a1fb9 not found: ID does not exist"
Nov 01 22:00:38.008873 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.008818    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 22:00:38.012662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.012635    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d5713372-openshift-default-5ddf58bc89-xmgvh"]
Nov 01 22:00:38.424920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.424845    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:00:38.425519 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.425502    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a7c800a0-789a-45b2-9048-ce6989b11ba6" containerName="istio-proxy"
Nov 01 22:00:38.425582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.425520    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c800a0-789a-45b2-9048-ce6989b11ba6" containerName="istio-proxy"
Nov 01 22:00:38.425645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.425636    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c800a0-789a-45b2-9048-ce6989b11ba6" containerName="istio-proxy"
Nov 01 22:00:38.430658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.430638    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.436577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.436552    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-f758d9-dockercfg-4djg7\""
Nov 01 22:00:38.436577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.436569    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-f758d9\""
Nov 01 22:00:38.441525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.441498    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:00:38.454388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.454004    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5kv\" (UniqueName: \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.454388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.454222    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.560768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.560715    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5kv\" (UniqueName: \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.560971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.560921    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.582318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.561915    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.643322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.641219    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5kv\" (UniqueName: \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\") pod \"payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.747183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.747155    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:38.871772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:38.871747    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c800a0-789a-45b2-9048-ce6989b11ba6" path="/var/lib/kubelet/pods/a7c800a0-789a-45b2-9048-ce6989b11ba6/volumes"
Nov 01 22:00:39.059420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:39.059390    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:00:39.060592 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:39.060566    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b65487_1cd9_4053_bb63_f50c185beb79.slice/crio-c29dc2351ebd9da5587bd4d7bf5833ca26f68218b5cea8eff35405bdfe42d9bf WatchSource:0}: Error finding container c29dc2351ebd9da5587bd4d7bf5833ca26f68218b5cea8eff35405bdfe42d9bf: Status 404 returned error can't find the container with id c29dc2351ebd9da5587bd4d7bf5833ca26f68218b5cea8eff35405bdfe42d9bf
Nov 01 22:00:39.993509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:39.993465    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" event={"ID":"c2b65487-1cd9-4053-bb63-f50c185beb79","Type":"ContainerStarted","Data":"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"}
Nov 01 22:00:39.993681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:39.993530    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" event={"ID":"c2b65487-1cd9-4053-bb63-f50c185beb79","Type":"ContainerStarted","Data":"c29dc2351ebd9da5587bd4d7bf5833ca26f68218b5cea8eff35405bdfe42d9bf"}
Nov 01 22:00:39.993681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:39.993562    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:40.054723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:40.054670    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" podStartSLOduration=2.054657671 podStartE2EDuration="2.054657671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:40.052627 +0000 UTC m=+943.308806764" watchObservedRunningTime="2026-11-01 22:00:40.054657671 +0000 UTC m=+943.310837435"
Nov 01 22:00:43.938978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:43.938941    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:00:43.944214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:43.944191    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:43.951226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:43.951197    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-f758d9-serving-cert\""
Nov 01 22:00:43.953831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:43.953796    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:00:43.991461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:43.991435    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:00:44.056640 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.056602    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.057412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.056716    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.057412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.056772    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hgw\" (UniqueName: \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.057412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.057044    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.158831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.158798    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.159031 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.159016    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.159110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.159068    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.159175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.159108    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hgw\" (UniqueName: \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.159775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.159748    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.161503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.161482    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.161599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.161535    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.171520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.171496    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hgw\" (UniqueName: \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\") pod \"maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.268929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.268897    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:44.578978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:44.578940    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:00:44.579322 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:00:44.579277    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db1449e_00fb_409c_8a1f_2a06ad3f94d5.slice/crio-fe3f4a7403c8f7596d4276fa47cb7d7119449b117b727df3050533396ec3bbf1 WatchSource:0}: Error finding container fe3f4a7403c8f7596d4276fa47cb7d7119449b117b727df3050533396ec3bbf1: Status 404 returned error can't find the container with id fe3f4a7403c8f7596d4276fa47cb7d7119449b117b727df3050533396ec3bbf1
Nov 01 22:00:45.036244 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:45.036209    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" event={"ID":"2db1449e-00fb-409c-8a1f-2a06ad3f94d5","Type":"ContainerStarted","Data":"fe3f4a7403c8f7596d4276fa47cb7d7119449b117b727df3050533396ec3bbf1"}
Nov 01 22:00:45.980321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:45.974074    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:00:46.077893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:46.074406    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" event={"ID":"2db1449e-00fb-409c-8a1f-2a06ad3f94d5","Type":"ContainerStarted","Data":"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"}
Nov 01 22:00:46.077893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:46.075535    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:00:46.128428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:46.128366    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" podStartSLOduration=2.663811271 podStartE2EDuration="3.128347161s" totalImagesPullingTime="464.53589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:00:46.101709172 +0000 UTC m=+949.357888937" watchObservedRunningTime="2026-11-01 22:00:46.128347161 +0000 UTC m=+949.384526925"
Nov 01 22:00:51.004209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:51.004180    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:00:53.136938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:00:53.136907    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:01:00.385344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.385308    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:01:00.391340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.391316    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.401587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.401563    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:01:00.405849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.405192    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.405849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.405261    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttlz\" (UniqueName: \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.458674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.458640    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-e2bf63-dockercfg-w9c8k\""
Nov 01 22:01:00.535458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.459510    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-e2bf63\""
Nov 01 22:01:00.535675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.509664    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.571319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.539409    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttlz\" (UniqueName: \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.571800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.510555    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.591610 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.591576    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttlz\" (UniqueName: \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\") pod \"payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:00.713247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:00.713158    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:01.007195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:01.007163    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:01:01.277717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:01.277636    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" event={"ID":"2d145ee6-4342-41e4-a95b-9365770eecf7","Type":"ContainerStarted","Data":"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"}
Nov 01 22:01:01.277717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:01.277686    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" event={"ID":"2d145ee6-4342-41e4-a95b-9365770eecf7","Type":"ContainerStarted","Data":"7770566054bf809af91970722540e8bea8632a4be32c9c9a70ee5da4e4857bf1"}
Nov 01 22:01:01.277897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:01.277818    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:01.298270 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:01.298220    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podStartSLOduration=1.298208153 podStartE2EDuration="1.298208153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:01.296611801 +0000 UTC m=+964.552791565" watchObservedRunningTime="2026-11-01 22:01:01.298208153 +0000 UTC m=+964.554387916"
Nov 01 22:01:03.675425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.675389    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:01:03.679840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.679821    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.692362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.692329    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:01:03.749319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.748622    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-2c5135-dockercfg-t8gkr\""
Nov 01 22:01:03.764331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.749746    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-2c5135\""
Nov 01 22:01:03.764331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.751926    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.764331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.752038    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmpc9\" (UniqueName: \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.863321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.859378    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.863321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.859495    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmpc9\" (UniqueName: \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.875713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.865641    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:03.890384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:03.889081    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmpc9\" (UniqueName: \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\") pod \"payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:04.000580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:04.000540    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:04.294700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:04.294667    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:01:04.296406 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:01:04.296375    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039aafea_77e9_4db4_9e2c_faf5abf0779f.slice/crio-7cdb1866d9fea11531a1a0a4eee1a013f6f2cf1acf452095fa7610d06eb84655 WatchSource:0}: Error finding container 7cdb1866d9fea11531a1a0a4eee1a013f6f2cf1acf452095fa7610d06eb84655: Status 404 returned error can't find the container with id 7cdb1866d9fea11531a1a0a4eee1a013f6f2cf1acf452095fa7610d06eb84655
Nov 01 22:01:04.824793 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:01:04.824757    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:01:05.308989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:05.308951    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" event={"ID":"039aafea-77e9-4db4-9e2c-faf5abf0779f","Type":"ContainerStarted","Data":"c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c"}
Nov 01 22:01:05.309163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:05.309029    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" event={"ID":"039aafea-77e9-4db4-9e2c-faf5abf0779f","Type":"ContainerStarted","Data":"7cdb1866d9fea11531a1a0a4eee1a013f6f2cf1acf452095fa7610d06eb84655"}
Nov 01 22:01:05.309208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:05.309179    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:05.328917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:05.328868    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podStartSLOduration=2.328856384 podStartE2EDuration="2.328856384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:05.328268647 +0000 UTC m=+968.584448411" watchObservedRunningTime="2026-11-01 22:01:05.328856384 +0000 UTC m=+968.585036147"
Nov 01 22:01:07.958872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:07.958840    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:01:07.968084 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:07.968054    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:07.973024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:07.972991    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:01:07.974474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:07.974445    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-e2bf63-serving-cert\""
Nov 01 22:01:08.025060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.025026    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.025211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.025074    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6lbx\" (UniqueName: \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.025254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.025221    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.025309 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.025283    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.126565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.126530    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.126774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.126578    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6lbx\" (UniqueName: \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.126774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.126652    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.126774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.126689    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.127420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.127391    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.129103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.129073    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.129210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.129193    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.139177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.139156    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6lbx\" (UniqueName: \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\") pod \"maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.298876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.298836    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:08.594730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:08.594697    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:01:09.346336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:09.346206    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" event={"ID":"a2c93af5-3488-4d83-960b-ead7d2f593e4","Type":"ContainerStarted","Data":"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"}
Nov 01 22:01:09.346336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:09.346265    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" event={"ID":"a2c93af5-3488-4d83-960b-ead7d2f593e4","Type":"ContainerStarted","Data":"7892d597b0675245468aa9ae44e1e0fed63dca26333971b5c6e915631858271a"}
Nov 01 22:01:09.346842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:09.346345    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:09.374864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:09.374820    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" podStartSLOduration=1.913811294 podStartE2EDuration="2.37480699s" totalImagesPullingTime="460.995696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:09.371633591 +0000 UTC m=+972.627813365" watchObservedRunningTime="2026-11-01 22:01:09.37480699 +0000 UTC m=+972.630986752"
Nov 01 22:01:12.289988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:12.289958    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:01:15.358487 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:15.358456    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:01:16.318408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:16.318379    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:01:29.643988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.643954    5845 generic.go:356] "Generic (PLEG): container finished" podID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerID="f484bfa3b61df0e2f9dd8eeac4379b86b596c53045c86708874d4c752c1d4564" exitCode=0
Nov 01 22:01:29.644444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.644002    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerDied","Data":"f484bfa3b61df0e2f9dd8eeac4379b86b596c53045c86708874d4c752c1d4564"}
Nov 01 22:01:29.644444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.644396    5845 scope.go:122] "RemoveContainer" containerID="f484bfa3b61df0e2f9dd8eeac4379b86b596c53045c86708874d4c752c1d4564"
Nov 01 22:01:29.645815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.645789    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:01:29.645877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.645854    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:01:29.645915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.645886    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:01:29.892161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.892059    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:01:29.898057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.896717    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:29.902027 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.901994    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:01:29.906502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.906476    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-21dd16de\""
Nov 01 22:01:29.945638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.945604    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:29.945808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:29.945747    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw27s\" (UniqueName: \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.081332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.077614    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw27s\" (UniqueName: \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.081332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.078019    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.081332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.079137    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.124630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.123336    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw27s\" (UniqueName: \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\") pod \"payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.229321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.221076    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.538762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.531070    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:01:30.681404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.681359    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" event={"ID":"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3","Type":"ContainerStarted","Data":"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"}
Nov 01 22:01:30.681872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.681426    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:30.681872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.681449    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" event={"ID":"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3","Type":"ContainerStarted","Data":"30dcc68bd32a528bdf10a95bc39cf4a74567754fb9840600b1429e2498cb8e78"}
Nov 01 22:01:30.683863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.683835    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerStarted","Data":"fce0365b7df0c2241ce80f02c638dfae77aa9a3c16e39eb5b7acaa791cc2f11d"}
Nov 01 22:01:30.701796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:30.701743    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" podStartSLOduration=1.7017292560000001 podStartE2EDuration="1.701729256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:30.700049024 +0000 UTC m=+993.956228788" watchObservedRunningTime="2026-11-01 22:01:30.701729256 +0000 UTC m=+993.957909019"
Nov 01 22:01:31.168320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:31.160657    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:01:31.168320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:31.161497    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:01:31.168320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:31.161548    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:01:32.169323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:32.168438    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:01:32.169323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:32.168502    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:01:33.179776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:33.167708    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:01:33.179776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:33.167812    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:01:34.154994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:34.153635    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:01:34.154994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:34.153691    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:01:34.817425 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:01:34.817391    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:01:35.158373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:35.158273    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:01:35.158737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:35.158701    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 01 22:01:35.952524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:35.952486    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:01:36.199664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:36.199619    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:01:36.729658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:36.729629    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:01:37.349588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.349556    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:01:37.355235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.355210    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.365392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.365361    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:01:37.366019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.365998    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-21dd16de-serving-cert\""
Nov 01 22:01:37.450653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.450622    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.450804 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.450686    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.450883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.450823    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.451009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.450990    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lxnq\" (UniqueName: \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.555429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.555381    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.555759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.555736    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.555952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.555936    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.556209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.556191    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lxnq\" (UniqueName: \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.556810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.556781    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.568677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.568574    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.568677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.568627    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.575029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.574995    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lxnq\" (UniqueName: \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\") pod \"maas-api-e2e-mt-21dd16de-85c75f9875-7rf54\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:37.678074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:37.677990    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:38.018858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:38.018812    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:01:38.744364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:38.744330    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" event={"ID":"270089a7-dc0d-4637-8228-88fc74e6bd3e","Type":"ContainerStarted","Data":"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"}
Nov 01 22:01:38.744813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:38.744382    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" event={"ID":"270089a7-dc0d-4637-8228-88fc74e6bd3e","Type":"ContainerStarted","Data":"96712799c8595e93d6e2fd22d4fc13d5205618927b9874c82eba19e1a5a42d1b"}
Nov 01 22:01:38.744813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:38.744411    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:01:38.768630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:38.768580    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" podStartSLOduration=1.199802408 podStartE2EDuration="1.768566823s" totalImagesPullingTime="568.764415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:01:38.763343636 +0000 UTC m=+1002.019523402" watchObservedRunningTime="2026-11-01 22:01:38.768566823 +0000 UTC m=+1002.024746586"
Nov 01 22:01:41.695471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:41.695440    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:01:44.757851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:01:44.757816    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:02:04.817901 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:02:04.817863    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:02:34.817792 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:02:34.817748    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:02:45.335210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.335177    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:02:45.339641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.339623    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.353674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.348589    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b1b5598d\""
Nov 01 22:02:45.353674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.349977    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b1b5598d-dockercfg-8gb7n\""
Nov 01 22:02:45.353674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.352492    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:02:45.359403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.359373    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpg8\" (UniqueName: \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.359590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.359468    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.477318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.474513    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpg8\" (UniqueName: \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.477318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.474618    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.527337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.523417    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.624373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.606511    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpg8\" (UniqueName: \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\") pod \"payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.665347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.665097    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:45.958477 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:45.958447    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:02:45.959754 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:02:45.959729    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8b6411_df1f_4bd7_85d2_7252dd526be1.slice/crio-33a80e858b555a486a885b87716a4be814b724b1d311ce089ad1d1b0cd4544b2 WatchSource:0}: Error finding container 33a80e858b555a486a885b87716a4be814b724b1d311ce089ad1d1b0cd4544b2: Status 404 returned error can't find the container with id 33a80e858b555a486a885b87716a4be814b724b1d311ce089ad1d1b0cd4544b2
Nov 01 22:02:46.621333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:46.621259    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" event={"ID":"af8b6411-df1f-4bd7-85d2-7252dd526be1","Type":"ContainerStarted","Data":"caffeec799f6d79015ddc7e969fb0bb761149cf83d88e0794be197d10c41eec4"}
Nov 01 22:02:46.621737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:46.621370    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" event={"ID":"af8b6411-df1f-4bd7-85d2-7252dd526be1","Type":"ContainerStarted","Data":"33a80e858b555a486a885b87716a4be814b724b1d311ce089ad1d1b0cd4544b2"}
Nov 01 22:02:46.621737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:46.621492    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:02:46.646472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:46.646395    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" podStartSLOduration=1.6463751869999999 podStartE2EDuration="1.646375187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:02:46.640910519 +0000 UTC m=+1069.897090286" watchObservedRunningTime="2026-11-01 22:02:46.646375187 +0000 UTC m=+1069.902554952"
Nov 01 22:02:53.475202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.475169    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:02:53.479926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.479903    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.497179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.497152    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b1b5598d-serving-cert\""
Nov 01 22:02:53.501547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.501519    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:02:53.553138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.553108    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.553282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.553232    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtjc\" (UniqueName: \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.553375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.553362    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.553428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.553416    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.659373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.657369    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.659787 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.659758    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtjc\" (UniqueName: \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.660482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.660449    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.660628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.660554    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.660842 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:02:53.660821    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-serving-cert: secret "maas-api-e2e-mt-b1b5598d-serving-cert" not found
Nov 01 22:02:53.660952 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:02:53.660884    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls podName:02ed5b06-67ce-49c8-be9b-bd246e5e4c8b nodeName:}" failed. No retries permitted until 2026-11-01 22:02:54.160862603 +0000 UTC m=+1077.417042344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls") pod "maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" (UID: "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b") : secret "maas-api-e2e-mt-b1b5598d-serving-cert" not found
Nov 01 22:02:53.661637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.661613    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.662392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.662364    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:53.673480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:53.673456    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtjc\" (UniqueName: \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:54.169825 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:54.169792    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:54.172535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:54.172506    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") pod \"maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:54.408959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:54.408932    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:54.697939 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:02:54.697893    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ed5b06_67ce_49c8_be9b_bd246e5e4c8b.slice/crio-c7b12205b87131777345c544c029a2514b4d9596575233d0d49d3fcf94f670aa WatchSource:0}: Error finding container c7b12205b87131777345c544c029a2514b4d9596575233d0d49d3fcf94f670aa: Status 404 returned error can't find the container with id c7b12205b87131777345c544c029a2514b4d9596575233d0d49d3fcf94f670aa
Nov 01 22:02:54.698685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:54.698048    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:02:55.689460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:55.689419    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" event={"ID":"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b","Type":"ContainerStarted","Data":"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"}
Nov 01 22:02:55.689651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:55.689484    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" event={"ID":"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b","Type":"ContainerStarted","Data":"c7b12205b87131777345c544c029a2514b4d9596575233d0d49d3fcf94f670aa"}
Nov 01 22:02:55.689789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:55.689769    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:02:55.710793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:55.710745    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" podStartSLOduration=2.196106775 podStartE2EDuration="2.710730665s" totalImagesPullingTime="514.62389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:02:55.709623638 +0000 UTC m=+1078.965803400" watchObservedRunningTime="2026-11-01 22:02:55.710730665 +0000 UTC m=+1078.966910429"
Nov 01 22:02:57.631639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:02:57.631608    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:03:01.703151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:01.703123    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:03:03.358858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:03.358821    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 22:03:03.359379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:03.359033    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy" containerID="cri-o://f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f" gracePeriod=30
Nov 01 22:03:04.859987 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:03:04.859946    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:03:08.656201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.656175    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 22:03:08.726703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.726665    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.726904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.726764    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.726974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.726908    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.726974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.726961    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727092    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727106    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:03:08.727201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727171    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727257    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727290    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727346    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\") pod \"2e835814-8b13-4f31-b151-3981d24740b2\" (UID: \"2e835814-8b13-4f31-b151-3981d24740b2\") "
Nov 01 22:03:08.727514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727367    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:03:08.727580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727555    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:03:08.727919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727855    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:03:08.728036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.727921    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:03:08.728207 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.728183    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.728418 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.728402    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.728543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.728529    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.728638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.728626    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e835814-8b13-4f31-b151-3981d24740b2-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.728725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.728713    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.730048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.730009    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:03:08.730151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.730131    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:03:08.730399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.730341    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:03:08.730399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.730339    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w" pod "2e835814-8b13-4f31-b151-3981d24740b2" (UID: "2e835814-8b13-4f31-b151-3981d24740b2"). InnerVolumeSpecName "kube-api-access-5869w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:03:08.830425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.830394    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.830425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.830421    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e835814-8b13-4f31-b151-3981d24740b2-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.830425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.830432    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5869w\" (UniqueName: \"kubernetes.io/projected/2e835814-8b13-4f31-b151-3981d24740b2-kube-api-access-5869w\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.830696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.830449    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e835814-8b13-4f31-b151-3981d24740b2-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:03:08.842686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.842649    5845 generic.go:356] "Generic (PLEG): container finished" podID="2e835814-8b13-4f31-b151-3981d24740b2" containerID="f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f" exitCode=0
Nov 01 22:03:08.842822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.842702    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"
Nov 01 22:03:08.842822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.842737    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerDied","Data":"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"}
Nov 01 22:03:08.842822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.842791    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl" event={"ID":"2e835814-8b13-4f31-b151-3981d24740b2","Type":"ContainerDied","Data":"d82bd3bcfa4a386867e33872598580133507023cf7e0b0c1113c8ede27441d52"}
Nov 01 22:03:08.842950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.842825    5845 scope.go:122] "RemoveContainer" containerID="f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"
Nov 01 22:03:08.903598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.901991    5845 scope.go:122] "RemoveContainer" containerID="f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"
Nov 01 22:03:08.907497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.907471    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 22:03:08.911138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.911100    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-78nhl"]
Nov 01 22:03:08.914659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.914636    5845 scope.go:122] "RemoveContainer" containerID="f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"
Nov 01 22:03:08.915028 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:03:08.915009    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f\": container with ID starting with f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f not found: ID does not exist" containerID="f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"
Nov 01 22:03:08.915089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.915037    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f"} err="failed to get container status \"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f\": rpc error: code = NotFound desc = could not find container \"f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f\": container with ID starting with f882fc28589896fdf8853fbd679538817614f698a96960f7b45736085c59fb9f not found: ID does not exist"
Nov 01 22:03:08.915089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.915057    5845 scope.go:122] "RemoveContainer" containerID="f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"
Nov 01 22:03:08.915378 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:03:08.915307    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f\": container with ID starting with f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f not found: ID does not exist" containerID="f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"
Nov 01 22:03:08.915378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:08.915336    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f"} err="failed to get container status \"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f\": rpc error: code = NotFound desc = could not find container \"f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f\": container with ID starting with f4ed5047f08837ef216d2e8498fb82b87137b73407daf4899134658d1e1fd98f not found: ID does not exist"
Nov 01 22:03:10.862327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:10.862260    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e835814-8b13-4f31-b151-3981d24740b2" path="/var/lib/kubelet/pods/2e835814-8b13-4f31-b151-3981d24740b2/volumes"
Nov 01 22:03:34.817756 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:03:34.817719    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:03:50.064859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:50.064824    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:03:51.495157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.495126    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:03:51.495927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.495910    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.495927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.495928    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.496073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.496065    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.496109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.496082    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.498732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.498717    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.510148 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.510126    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:03:51.510841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.510820    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.510841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.510841    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e835814-8b13-4f31-b151-3981d24740b2" containerName="istio-proxy"
Nov 01 22:03:51.512578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.512552    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a19582\""
Nov 01 22:03:51.514007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.513991    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.515291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.515271    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:03:51.524717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.523618    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a19582-dockercfg-vdkjt\""
Nov 01 22:03:51.532124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.532096    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:03:51.594037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.594006    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmkw\" (UniqueName: \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.594208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.594048    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zrd\" (UniqueName: \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.594208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.594071    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.594386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.594363    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.709319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.709180    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.709515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.709432    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmkw\" (UniqueName: \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.709515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.709467    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zrd\" (UniqueName: \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.709515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.709491    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.713318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.710544    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.713318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.711269    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.751471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.751339    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zrd\" (UniqueName: \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\") pod \"payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.753432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.753361    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmkw\" (UniqueName: \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\") pod \"payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:51.820678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.819928    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:51.837063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:51.837033    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:52.187580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.187551    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:03:52.188967 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:03:52.188941    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155bd7a0_52c5_4993_bb86_6654ff095133.slice/crio-de2513802195b8ac7a6aca7f020bc322c14f05dbcb783e5fcb0677cdbcd4ff27 WatchSource:0}: Error finding container de2513802195b8ac7a6aca7f020bc322c14f05dbcb783e5fcb0677cdbcd4ff27: Status 404 returned error can't find the container with id de2513802195b8ac7a6aca7f020bc322c14f05dbcb783e5fcb0677cdbcd4ff27
Nov 01 22:03:52.208681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.208654    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:03:52.208989 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:03:52.208974    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0866b18f_b24e_4169_984b_f01004ce6e3d.slice/crio-99bc9ba5db5e08951234e0e6b23f682e16418c55e39516f6181fa097d6f374ac WatchSource:0}: Error finding container 99bc9ba5db5e08951234e0e6b23f682e16418c55e39516f6181fa097d6f374ac: Status 404 returned error can't find the container with id 99bc9ba5db5e08951234e0e6b23f682e16418c55e39516f6181fa097d6f374ac
Nov 01 22:03:52.314788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.314752    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" event={"ID":"0866b18f-b24e-4169-984b-f01004ce6e3d","Type":"ContainerStarted","Data":"3505ac835f3b789af092060d9ce5036da11c2d68f3b171132cda0e25155dd636"}
Nov 01 22:03:52.314936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.314820    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:03:52.314936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.314839    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" event={"ID":"0866b18f-b24e-4169-984b-f01004ce6e3d","Type":"ContainerStarted","Data":"99bc9ba5db5e08951234e0e6b23f682e16418c55e39516f6181fa097d6f374ac"}
Nov 01 22:03:52.316501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.316475    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" event={"ID":"155bd7a0-52c5-4993-bb86-6654ff095133","Type":"ContainerStarted","Data":"203b0fd6c9d6115ed684b69f2f6bd462158eb07ea5f2c3fa3e9c0a70d863515e"}
Nov 01 22:03:52.316624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.316521    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" event={"ID":"155bd7a0-52c5-4993-bb86-6654ff095133","Type":"ContainerStarted","Data":"de2513802195b8ac7a6aca7f020bc322c14f05dbcb783e5fcb0677cdbcd4ff27"}
Nov 01 22:03:52.316624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.316564    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:03:52.333413 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.333266    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" podStartSLOduration=1.333251996 podStartE2EDuration="1.333251996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:52.332833324 +0000 UTC m=+1135.589013088" watchObservedRunningTime="2026-11-01 22:03:52.333251996 +0000 UTC m=+1135.589431760"
Nov 01 22:03:52.356385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:52.356331    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" podStartSLOduration=1.356292832 podStartE2EDuration="1.356292832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:03:52.353600881 +0000 UTC m=+1135.609780663" watchObservedRunningTime="2026-11-01 22:03:52.356292832 +0000 UTC m=+1135.612472598"
Nov 01 22:03:55.065863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:03:55.065767    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing" containerID="cri-o://a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d" gracePeriod=25
Nov 01 22:04:00.833281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.833248    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:04:00.841470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.840850    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:00.852448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.852422    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:04:00.856044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.856014    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a19582-serving-cert\""
Nov 01 22:04:00.979860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.979820    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvtf\" (UniqueName: \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:00.980687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.980587    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:00.981238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.981216    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:00.982784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:00.982757    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.084959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.084877    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.085110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.085018    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvtf\" (UniqueName: \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.085110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.085103    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.085289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.085271    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.085696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.085673    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.087745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.087718    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.087745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.087741    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.100035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.100008    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvtf\" (UniqueName: \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\") pod \"maas-api-e2e-api-a-a19582-6487545c6c-m49fx\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.162395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.162368    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:01.458604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.458577    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:04:01.462055 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:01.462022    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08562f88_02b9_4aab_b4bd_0ad93aaa83d6.slice/crio-f2da6781f27a15eef89da0dfce1223766c337ab71272ceaf11d75b4025eec4c9 WatchSource:0}: Error finding container f2da6781f27a15eef89da0dfce1223766c337ab71272ceaf11d75b4025eec4c9: Status 404 returned error can't find the container with id f2da6781f27a15eef89da0dfce1223766c337ab71272ceaf11d75b4025eec4c9
Nov 01 22:04:01.741321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:01.733185    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:04:02.403858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:02.403763    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" event={"ID":"08562f88-02b9-4aab-b4bd-0ad93aaa83d6","Type":"ContainerStarted","Data":"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"}
Nov 01 22:04:02.403858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:02.403825    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" event={"ID":"08562f88-02b9-4aab-b4bd-0ad93aaa83d6","Type":"ContainerStarted","Data":"f2da6781f27a15eef89da0dfce1223766c337ab71272ceaf11d75b4025eec4c9"}
Nov 01 22:04:02.403858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:02.403849    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:02.427470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:02.427423    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" podStartSLOduration=1.956422651 podStartE2EDuration="2.427409503s" totalImagesPullingTime="470.986852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:02.424441926 +0000 UTC m=+1145.680621687" watchObservedRunningTime="2026-11-01 22:04:02.427409503 +0000 UTC m=+1145.683589266"
Nov 01 22:04:03.326715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:03.326686    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:04:03.328380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:03.328349    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:04:04.819071 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:04:04.819023    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:04:07.981793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:07.981757    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:04:08.168045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.168006    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:04:08.168314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.168210    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.179375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.179193    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edd32a\""
Nov 01 22:04:08.192340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.192317    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-edd32a-dockercfg-kxn95\""
Nov 01 22:04:08.274217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.274127    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.274784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.274752    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbjr\" (UniqueName: \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.377758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.377718    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbjr\" (UniqueName: \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.378209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.378180    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.379208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.379178    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.393977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.393859    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbjr\" (UniqueName: \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\") pod \"payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.426796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.426757    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:04:08.508389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.504717    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:08.863130 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:08.863094    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cff68f7_cebc_4677_9548_fd42f95cbcf0.slice/crio-4f333eff52ca95f321adaf9f118d4bd2fd24d67cb9406efae3e7a8feb3e8f331 WatchSource:0}: Error finding container 4f333eff52ca95f321adaf9f118d4bd2fd24d67cb9406efae3e7a8feb3e8f331: Status 404 returned error can't find the container with id 4f333eff52ca95f321adaf9f118d4bd2fd24d67cb9406efae3e7a8feb3e8f331
Nov 01 22:04:08.865177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:08.865143    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:04:09.601139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:09.601100    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" event={"ID":"4cff68f7-cebc-4677-9548-fd42f95cbcf0","Type":"ContainerStarted","Data":"3afb26c5936811d30fb4ed3327953b6e2a2a8c3e204ce44fe0e495c23b85da93"}
Nov 01 22:04:09.601527 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:09.601152    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" event={"ID":"4cff68f7-cebc-4677-9548-fd42f95cbcf0","Type":"ContainerStarted","Data":"4f333eff52ca95f321adaf9f118d4bd2fd24d67cb9406efae3e7a8feb3e8f331"}
Nov 01 22:04:09.601527 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:09.601175    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:09.621504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:09.621464    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" podStartSLOduration=2.621450689 podStartE2EDuration="2.621450689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:09.621055926 +0000 UTC m=+1152.877235690" watchObservedRunningTime="2026-11-01 22:04:09.621450689 +0000 UTC m=+1152.877630451"
Nov 01 22:04:11.695243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:11.695204    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:04:15.402461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.402433    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:04:15.515791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.515760    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\") pod \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") "
Nov 01 22:04:15.515961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.515878    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\" (UniqueName: \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\") pod \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\" (UID: \"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3\") "
Nov 01 22:04:15.516141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.516116    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume" pod "f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" (UID: "f0eef0ed-0d73-4ea3-bf42-921127c1f1b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:04:15.516721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.516686    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:15.518009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.517986    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s" pod "f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" (UID: "f0eef0ed-0d73-4ea3-bf42-921127c1f1b3"). InnerVolumeSpecName "kube-api-access-rw27s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:04:15.617847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.617773    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw27s\" (UniqueName: \"kubernetes.io/projected/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3-kube-api-access-rw27s\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:15.687200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.687114    5845 generic.go:356] "Generic (PLEG): container finished" podID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerID="a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d" exitCode=0
Nov 01 22:04:15.687364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.687226    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"
Nov 01 22:04:15.687920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.687403    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" event={"ID":"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3","Type":"ContainerDied","Data":"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"}
Nov 01 22:04:15.687920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.687454    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln" event={"ID":"f0eef0ed-0d73-4ea3-bf42-921127c1f1b3","Type":"ContainerDied","Data":"30dcc68bd32a528bdf10a95bc39cf4a74567754fb9840600b1429e2498cb8e78"}
Nov 01 22:04:15.687920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.687477    5845 scope.go:122] "RemoveContainer" containerID="a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"
Nov 01 22:04:15.698160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.698143    5845 scope.go:122] "RemoveContainer" containerID="a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"
Nov 01 22:04:15.698451 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:04:15.698418    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d\": container with ID starting with a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d not found: ID does not exist" containerID="a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"
Nov 01 22:04:15.698515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.698457    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d"} err="failed to get container status \"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d\": rpc error: code = NotFound desc = could not find container \"a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d\": container with ID starting with a07a0cd286d8d478999bebc4844f097704769224dd3a0f7bcfc19fbe2bc2a68d not found: ID does not exist"
Nov 01 22:04:15.713640 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.713615    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:04:15.717532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:15.717510    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-21dd16de-568bb59d5f-4djln"]
Nov 01 22:04:16.862845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:16.862803    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" path="/var/lib/kubelet/pods/f0eef0ed-0d73-4ea3-bf42-921127c1f1b3/volumes"
Nov 01 22:04:17.309195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.309155    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:04:17.309922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.309906    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing"
Nov 01 22:04:17.309995 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.309923    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing"
Nov 01 22:04:17.310072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.310063    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0eef0ed-0d73-4ea3-bf42-921127c1f1b3" containerName="payload-pre-processing"
Nov 01 22:04:17.315141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.315111    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.319588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.319563    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-004180-dockercfg-8qcxg\""
Nov 01 22:04:17.322964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.322941    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:04:17.330976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.330950    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-004180\""
Nov 01 22:04:17.366477 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.366436    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:04:17.370906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.370890    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.384511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.384488    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:04:17.495319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.461054    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55p4x\" (UniqueName: \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.537069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.537037    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.650174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.649910    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.651271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.651243    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.651509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.651260    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.651898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.651827    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5mk\" (UniqueName: \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.652019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.651995    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55p4x\" (UniqueName: \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.667347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.666121    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55p4x\" (UniqueName: \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\") pod \"payload-pre-processing-e2e-api-b-004180-845f96955-lw94h\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.754021 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.753985    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.754254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.754203    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5mk\" (UniqueName: \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.755002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.754972    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.766911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.766882    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5mk\" (UniqueName: \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\") pod \"payload-processing-e2e-api-b-004180-548d486474-4m87j\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") " pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:17.932187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.932101    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:17.994591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:17.994208    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:18.427698 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:18.427667    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ecf857_b357_42b1_a7be_57f7bfbbdd08.slice/crio-99ce1f57dad44dd54f8552d972ace0dd0a62938eb5fdb42e6012d168847daee9 WatchSource:0}: Error finding container 99ce1f57dad44dd54f8552d972ace0dd0a62938eb5fdb42e6012d168847daee9: Status 404 returned error can't find the container with id 99ce1f57dad44dd54f8552d972ace0dd0a62938eb5fdb42e6012d168847daee9
Nov 01 22:04:18.428198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.428172    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:04:18.510837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.510805    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:04:18.511581 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:18.511434    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ecbada_df58_4383_89ef_5d3cf69a0f65.slice/crio-e06e5d996eb6935f9f2005f68e6cf7308b686ff8dac4f436efb02401f957fc72 WatchSource:0}: Error finding container e06e5d996eb6935f9f2005f68e6cf7308b686ff8dac4f436efb02401f957fc72: Status 404 returned error can't find the container with id e06e5d996eb6935f9f2005f68e6cf7308b686ff8dac4f436efb02401f957fc72
Nov 01 22:04:18.716535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.716500    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" event={"ID":"d5ecf857-b357-42b1-a7be-57f7bfbbdd08","Type":"ContainerStarted","Data":"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"}
Nov 01 22:04:18.716730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.716559    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" event={"ID":"d5ecf857-b357-42b1-a7be-57f7bfbbdd08","Type":"ContainerStarted","Data":"99ce1f57dad44dd54f8552d972ace0dd0a62938eb5fdb42e6012d168847daee9"}
Nov 01 22:04:18.716730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.716585    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:18.718390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.718359    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" event={"ID":"96ecbada-df58-4383-89ef-5d3cf69a0f65","Type":"ContainerStarted","Data":"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"}
Nov 01 22:04:18.718506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.718427    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:18.718506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.718446    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" event={"ID":"96ecbada-df58-4383-89ef-5d3cf69a0f65","Type":"ContainerStarted","Data":"e06e5d996eb6935f9f2005f68e6cf7308b686ff8dac4f436efb02401f957fc72"}
Nov 01 22:04:18.736017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.735971    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" podStartSLOduration=1.735957886 podStartE2EDuration="1.735957886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:18.733648802 +0000 UTC m=+1161.989828564" watchObservedRunningTime="2026-11-01 22:04:18.735957886 +0000 UTC m=+1161.992137693"
Nov 01 22:04:18.759769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:18.759725    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" podStartSLOduration=1.759712499 podStartE2EDuration="1.759712499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:18.755729757 +0000 UTC m=+1162.011909520" watchObservedRunningTime="2026-11-01 22:04:18.759712499 +0000 UTC m=+1162.015892280"
Nov 01 22:04:20.238945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.238890    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:04:20.239444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.239193    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" podUID="270089a7-dc0d-4637-8228-88fc74e6bd3e" containerName="maas-api" containerID="cri-o://288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64" gracePeriod=30
Nov 01 22:04:20.698359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.697393    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:04:20.703322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.699658    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:04:20.747582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.747544    5845 generic.go:356] "Generic (PLEG): container finished" podID="270089a7-dc0d-4637-8228-88fc74e6bd3e" containerID="288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64" exitCode=0
Nov 01 22:04:20.747856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.747599    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"
Nov 01 22:04:20.747856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.747628    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" event={"ID":"270089a7-dc0d-4637-8228-88fc74e6bd3e","Type":"ContainerDied","Data":"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"}
Nov 01 22:04:20.747856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.747673    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54" event={"ID":"270089a7-dc0d-4637-8228-88fc74e6bd3e","Type":"ContainerDied","Data":"96712799c8595e93d6e2fd22d4fc13d5205618927b9874c82eba19e1a5a42d1b"}
Nov 01 22:04:20.747856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.747710    5845 scope.go:122] "RemoveContainer" containerID="288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"
Nov 01 22:04:20.761862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.761825    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\") pod \"270089a7-dc0d-4637-8228-88fc74e6bd3e\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") "
Nov 01 22:04:20.762038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.762016    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\" (UniqueName: \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\") pod \"270089a7-dc0d-4637-8228-88fc74e6bd3e\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") "
Nov 01 22:04:20.762116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.762090    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\") pod \"270089a7-dc0d-4637-8228-88fc74e6bd3e\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") "
Nov 01 22:04:20.763253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.762360    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\") pod \"270089a7-dc0d-4637-8228-88fc74e6bd3e\" (UID: \"270089a7-dc0d-4637-8228-88fc74e6bd3e\") "
Nov 01 22:04:20.765903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.765880    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca" pod "270089a7-dc0d-4637-8228-88fc74e6bd3e" (UID: "270089a7-dc0d-4637-8228-88fc74e6bd3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:04:20.770574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.770553    5845 scope.go:122] "RemoveContainer" containerID="288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"
Nov 01 22:04:20.771171 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:04:20.771146    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64\": container with ID starting with 288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64 not found: ID does not exist" containerID="288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"
Nov 01 22:04:20.771335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.771311    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64"} err="failed to get container status \"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64\": rpc error: code = NotFound desc = could not find container \"288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64\": container with ID starting with 288caa086b0189df77ca1cc8e081bf94190e2c2e6b99049387cde1b475b99e64 not found: ID does not exist"
Nov 01 22:04:20.779658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.776863    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls" pod "270089a7-dc0d-4637-8228-88fc74e6bd3e" (UID: "270089a7-dc0d-4637-8228-88fc74e6bd3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:04:20.792044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.792009    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq" pod "270089a7-dc0d-4637-8228-88fc74e6bd3e" (UID: "270089a7-dc0d-4637-8228-88fc74e6bd3e"). InnerVolumeSpecName "kube-api-access-6lxnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:04:20.793232 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.793208    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs" pod "270089a7-dc0d-4637-8228-88fc74e6bd3e" (UID: "270089a7-dc0d-4637-8228-88fc74e6bd3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:04:20.865994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.865910    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:20.865994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.865944    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/270089a7-dc0d-4637-8228-88fc74e6bd3e-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:20.865994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.865961    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lxnq\" (UniqueName: \"kubernetes.io/projected/270089a7-dc0d-4637-8228-88fc74e6bd3e-kube-api-access-6lxnq\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:20.865994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:20.865977    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/270089a7-dc0d-4637-8228-88fc74e6bd3e-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:21.067458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.067414    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:04:21.071067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.071039    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-21dd16de-85c75f9875-7rf54"]
Nov 01 22:04:21.637886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.637848    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:04:21.638955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.638933    5845 cpu_manager.go:414] "containerMap: removing container" podUID="270089a7-dc0d-4637-8228-88fc74e6bd3e" containerName="maas-api"
Nov 01 22:04:21.638955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.638954    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270089a7-dc0d-4637-8228-88fc74e6bd3e" containerName="maas-api"
Nov 01 22:04:21.639149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.639084    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="270089a7-dc0d-4637-8228-88fc74e6bd3e" containerName="maas-api"
Nov 01 22:04:21.644146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.644126    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.650828 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.650803    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:04:21.652052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.652027    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0390e1-openshift-default-dockercfg-nx2rx\""
Nov 01 22:04:21.683251 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683218    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683257    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683402    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683444    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683538    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683701    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683760    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk7c\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683888    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.683937 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.683927    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.827321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.792812    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886261    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886407    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886442    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886490    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886513    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886607    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886782    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.886859    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk7c\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.887944    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.888214    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.888470    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.889290    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.909069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.889991    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.920050 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.915606    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk7c\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.970776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.959454    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:21.986322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:21.983618    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:22.004969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.003477    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\") pod \"e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") " pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:22.269874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.269844    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:22.561997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.561950    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:04:22.563708 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:22.563662    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd205146_f5cd_4603_aa2a_cf5384811873.slice/crio-710c058c8ffdcf1aab7c477aacdd22323257706329a68497b0963c4a594f0690 WatchSource:0}: Error finding container 710c058c8ffdcf1aab7c477aacdd22323257706329a68497b0963c4a594f0690: Status 404 returned error can't find the container with id 710c058c8ffdcf1aab7c477aacdd22323257706329a68497b0963c4a594f0690
Nov 01 22:04:22.567513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.567477    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:04:22.567627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.567574    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:04:22.567627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.567612    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:04:22.879146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:22.879071    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270089a7-dc0d-4637-8228-88fc74e6bd3e" path="/var/lib/kubelet/pods/270089a7-dc0d-4637-8228-88fc74e6bd3e/volumes"
Nov 01 22:04:23.013262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:23.013218    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerStarted","Data":"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"}
Nov 01 22:04:23.016586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:23.016557    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerStarted","Data":"710c058c8ffdcf1aab7c477aacdd22323257706329a68497b0963c4a594f0690"}
Nov 01 22:04:23.055598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:23.055531    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podStartSLOduration=2.055510346 podStartE2EDuration="2.055510346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:23.053008092 +0000 UTC m=+1166.309187873" watchObservedRunningTime="2026-11-01 22:04:23.055510346 +0000 UTC m=+1166.311690107"
Nov 01 22:04:23.270410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:23.270372    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:23.277235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:23.277209    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:24.019938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:24.019902    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:24.020963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:24.020942    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:04:29.729851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:29.729817    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:04:29.730289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:29.730230    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:04:34.829760 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:04:34.829723    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:04:47.328452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.325681    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:04:47.331537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.331511    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.352567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.352530    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-702002\""
Nov 01 22:04:47.492323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.468005    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-702002-dockercfg-bg6pq\""
Nov 01 22:04:47.492551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.486671    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-5wxxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:04:47.500323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.492724    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:04:47.572376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.561353    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:04:47.612796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.572845    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9r5d\" (UniqueName: \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.612796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.573271    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.612796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.587459    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-7bg9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:04:47.612796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.587534    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:04:47.676506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.676462    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.679320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.679270    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.682768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.682744    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9r5d\" (UniqueName: \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.740322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.740255    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9r5d\" (UniqueName: \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\") pod \"payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:47.976801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:47.976763    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:48.261393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:48.261364    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:04:48.261854 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:48.261826    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fe8380_5468_40d1_b793_28069e2d90ab.slice/crio-8534dd09c16743c41cd40bd6876b6d85abb1356b8380a45c81124fcb186ecb3e WatchSource:0}: Error finding container 8534dd09c16743c41cd40bd6876b6d85abb1356b8380a45c81124fcb186ecb3e: Status 404 returned error can't find the container with id 8534dd09c16743c41cd40bd6876b6d85abb1356b8380a45c81124fcb186ecb3e
Nov 01 22:04:48.681204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:48.681107    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" event={"ID":"51fe8380-5468-40d1-b793-28069e2d90ab","Type":"ContainerStarted","Data":"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"}
Nov 01 22:04:48.681204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:48.681175    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:48.681204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:48.681198    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" event={"ID":"51fe8380-5468-40d1-b793-28069e2d90ab","Type":"ContainerStarted","Data":"8534dd09c16743c41cd40bd6876b6d85abb1356b8380a45c81124fcb186ecb3e"}
Nov 01 22:04:48.704417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:48.704359    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podStartSLOduration=1.704341568 podStartE2EDuration="1.704341568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:48.699818167 +0000 UTC m=+1191.955997931" watchObservedRunningTime="2026-11-01 22:04:48.704341568 +0000 UTC m=+1191.960521333"
Nov 01 22:04:49.911152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.911117    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:04:49.915855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.915837    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:49.921548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.921521    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3577a0\""
Nov 01 22:04:49.921689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.921613    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3577a0-dockercfg-tsttb\""
Nov 01 22:04:49.926621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.926597    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:04:49.951651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.951626    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:04:49.956160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.956143    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:49.970799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:49.970771    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:04:50.036747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.032261    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcghx\" (UniqueName: \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.037083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.037049    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfswm\" (UniqueName: \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.061331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.044683    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.061774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.061746    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.167799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.165933    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcghx\" (UniqueName: \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.167799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.165993    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfswm\" (UniqueName: \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.167799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.166045    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.167799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.166175    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.167799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.167316    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.169186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.169139    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.190774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.184368    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcghx\" (UniqueName: \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\") pod \"payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.215774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.209852    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfswm\" (UniqueName: \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\") pod \"payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.233884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.233853    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.278585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.278553    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.592163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.592127    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:04:50.593391 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:50.593361    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda8271f_8236_45c3_9c46_32f89a983da0.slice/crio-358abad0f0b4748c4171c2bb5133eca5bed127327d9cefe41a64463c7d3f832f WatchSource:0}: Error finding container 358abad0f0b4748c4171c2bb5133eca5bed127327d9cefe41a64463c7d3f832f: Status 404 returned error can't find the container with id 358abad0f0b4748c4171c2bb5133eca5bed127327d9cefe41a64463c7d3f832f
Nov 01 22:04:50.598465 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:04:50.598375    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d44d0c_c985_4f79_8b90_5a3c7b5d190a.slice/crio-eef9db785d58f2f0cce02221fd276dc8bbb49a3e77605850b166e98d4f31d8f1 WatchSource:0}: Error finding container eef9db785d58f2f0cce02221fd276dc8bbb49a3e77605850b166e98d4f31d8f1: Status 404 returned error can't find the container with id eef9db785d58f2f0cce02221fd276dc8bbb49a3e77605850b166e98d4f31d8f1
Nov 01 22:04:50.598536 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.598467    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:04:50.698001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.697974    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" event={"ID":"50d44d0c-c985-4f79-8b90-5a3c7b5d190a","Type":"ContainerStarted","Data":"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"}
Nov 01 22:04:50.698130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.698038    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:04:50.698130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.698064    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" event={"ID":"50d44d0c-c985-4f79-8b90-5a3c7b5d190a","Type":"ContainerStarted","Data":"eef9db785d58f2f0cce02221fd276dc8bbb49a3e77605850b166e98d4f31d8f1"}
Nov 01 22:04:50.699506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.699482    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" event={"ID":"dda8271f-8236-45c3-9c46-32f89a983da0","Type":"ContainerStarted","Data":"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"}
Nov 01 22:04:50.699594 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.699522    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" event={"ID":"dda8271f-8236-45c3-9c46-32f89a983da0","Type":"ContainerStarted","Data":"358abad0f0b4748c4171c2bb5133eca5bed127327d9cefe41a64463c7d3f832f"}
Nov 01 22:04:50.699594 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.699567    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:04:50.717333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.717254    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" podStartSLOduration=1.717241339 podStartE2EDuration="1.717241339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:50.716190158 +0000 UTC m=+1193.972369922" watchObservedRunningTime="2026-11-01 22:04:50.717241339 +0000 UTC m=+1193.973421100"
Nov 01 22:04:50.735958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:50.735908    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" podStartSLOduration=1.735892266 podStartE2EDuration="1.735892266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:04:50.733654717 +0000 UTC m=+1193.989834491" watchObservedRunningTime="2026-11-01 22:04:50.735892266 +0000 UTC m=+1193.992072032"
Nov 01 22:04:51.482447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:51.482411    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:04:56.485320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:56.484392    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" podUID="af8b6411-df1f-4bd7-85d2-7252dd526be1" containerName="payload-pre-processing" containerID="cri-o://caffeec799f6d79015ddc7e969fb0bb761149cf83d88e0794be197d10c41eec4" gracePeriod=25
Nov 01 22:04:56.755869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:56.755835    5845 generic.go:356] "Generic (PLEG): container finished" podID="af8b6411-df1f-4bd7-85d2-7252dd526be1" containerID="caffeec799f6d79015ddc7e969fb0bb761149cf83d88e0794be197d10c41eec4" exitCode=0
Nov 01 22:04:56.756205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:56.756179    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" event={"ID":"af8b6411-df1f-4bd7-85d2-7252dd526be1","Type":"ContainerDied","Data":"caffeec799f6d79015ddc7e969fb0bb761149cf83d88e0794be197d10c41eec4"}
Nov 01 22:04:56.813860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:56.813266    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:04:57.005521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.005490    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\") pod \"af8b6411-df1f-4bd7-85d2-7252dd526be1\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") "
Nov 01 22:04:57.005744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.005549    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\" (UniqueName: \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\") pod \"af8b6411-df1f-4bd7-85d2-7252dd526be1\" (UID: \"af8b6411-df1f-4bd7-85d2-7252dd526be1\") "
Nov 01 22:04:57.005927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.005885    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume" pod "af8b6411-df1f-4bd7-85d2-7252dd526be1" (UID: "af8b6411-df1f-4bd7-85d2-7252dd526be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:04:57.006824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.006794    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af8b6411-df1f-4bd7-85d2-7252dd526be1-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:57.008503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.008476    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8" pod "af8b6411-df1f-4bd7-85d2-7252dd526be1" (UID: "af8b6411-df1f-4bd7-85d2-7252dd526be1"). InnerVolumeSpecName "kube-api-access-ldpg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:04:57.048813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.048789    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:04:57.052057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.052035    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:04:57.108691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.108662    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldpg8\" (UniqueName: \"kubernetes.io/projected/af8b6411-df1f-4bd7-85d2-7252dd526be1-kube-api-access-ldpg8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:57.767727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.767690    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"
Nov 01 22:04:57.768260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.767728    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x" event={"ID":"af8b6411-df1f-4bd7-85d2-7252dd526be1","Type":"ContainerDied","Data":"33a80e858b555a486a885b87716a4be814b724b1d311ce089ad1d1b0cd4544b2"}
Nov 01 22:04:57.768260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.767796    5845 scope.go:122] "RemoveContainer" containerID="caffeec799f6d79015ddc7e969fb0bb761149cf83d88e0794be197d10c41eec4"
Nov 01 22:04:57.794676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.794652    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:04:57.802769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:57.802744    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b1b5598d-5779c6d5c5-lrj7x"]
Nov 01 22:04:58.863021 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:58.862978    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8b6411-df1f-4bd7-85d2-7252dd526be1" path="/var/lib/kubelet/pods/af8b6411-df1f-4bd7-85d2-7252dd526be1/volumes"
Nov 01 22:04:58.913318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:58.907957    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:04:58.913318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:58.908213    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" podUID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" containerName="maas-api" containerID="cri-o://9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f" gracePeriod=30
Nov 01 22:04:59.434334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.434280    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:04:59.496472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.496431    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\") pod \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") "
Nov 01 22:04:59.496699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.496681    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\" (UniqueName: \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\") pod \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") "
Nov 01 22:04:59.496799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.496712    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") pod \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") "
Nov 01 22:04:59.496799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.496761    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca" pod "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" (UID: "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:04:59.496924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.496859    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\") pod \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\" (UID: \"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b\") "
Nov 01 22:04:59.497550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.497527    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:59.498874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.498841    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs" pod "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" (UID: "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:04:59.499219 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.499190    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls" pod "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" (UID: "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:04:59.499350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.499275    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc" pod "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" (UID: "02ed5b06-67ce-49c8-be9b-bd246e5e4c8b"). InnerVolumeSpecName "kube-api-access-4gtjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:04:59.599039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.599005    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:59.599039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.599035    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtjc\" (UniqueName: \"kubernetes.io/projected/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-kube-api-access-4gtjc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:59.599039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.599046    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:04:59.693567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.693481    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:04:59.784112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.784069    5845 generic.go:356] "Generic (PLEG): container finished" podID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" containerID="9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f" exitCode=0
Nov 01 22:04:59.784315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.784120    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"
Nov 01 22:04:59.784315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.784146    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" event={"ID":"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b","Type":"ContainerDied","Data":"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"}
Nov 01 22:04:59.784315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.784212    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq" event={"ID":"02ed5b06-67ce-49c8-be9b-bd246e5e4c8b","Type":"ContainerDied","Data":"c7b12205b87131777345c544c029a2514b4d9596575233d0d49d3fcf94f670aa"}
Nov 01 22:04:59.784315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.784241    5845 scope.go:122] "RemoveContainer" containerID="9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"
Nov 01 22:04:59.797442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.797419    5845 scope.go:122] "RemoveContainer" containerID="9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"
Nov 01 22:04:59.797742 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:04:59.797722    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f\": container with ID starting with 9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f not found: ID does not exist" containerID="9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"
Nov 01 22:04:59.797801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.797752    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f"} err="failed to get container status \"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f\": rpc error: code = NotFound desc = could not find container \"9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f\": container with ID starting with 9ce9b0a98089b420db0f257154f8e5564e5d855d64df78361d96be25b897665f not found: ID does not exist"
Nov 01 22:04:59.813032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.813000    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:04:59.821865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:04:59.821835    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b1b5598d-989bbb4-lnkcq"]
Nov 01 22:05:00.865875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:00.865843    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" path="/var/lib/kubelet/pods/02ed5b06-67ce-49c8-be9b-bd246e5e4c8b/volumes"
Nov 01 22:05:01.709561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:01.709533    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:05:01.709724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:01.709601    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:05:03.687511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:03.687475    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 22:05:03.687947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:03.687777    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy" containerID="cri-o://56eb684e1be9c4dfcf8253cdac6b57c6d2b2b9dac3a16d7260ca60f16599ea29" gracePeriod=30
Nov 01 22:05:03.757525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:03.757495    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 22:05:03.757715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:03.757695    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy" containerID="cri-o://4b1b31a77dfb3296f07f0d419d7410f8dab671d24ef9fbb935df23dd48c84ad5" gracePeriod=30
Nov 01 22:05:04.817417 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:05:04.817376    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:05:07.244243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.244207    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:05:07.245410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245383    5845 cpu_manager.go:414] "containerMap: removing container" podUID="af8b6411-df1f-4bd7-85d2-7252dd526be1" containerName="payload-pre-processing"
Nov 01 22:05:07.245410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245409    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8b6411-df1f-4bd7-85d2-7252dd526be1" containerName="payload-pre-processing"
Nov 01 22:05:07.245619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245422    5845 cpu_manager.go:414] "containerMap: removing container" podUID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" containerName="maas-api"
Nov 01 22:05:07.245619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245429    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" containerName="maas-api"
Nov 01 22:05:07.245717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245616    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8b6411-df1f-4bd7-85d2-7252dd526be1" containerName="payload-pre-processing"
Nov 01 22:05:07.245717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.245630    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ed5b06-67ce-49c8-be9b-bd246e5e4c8b" containerName="maas-api"
Nov 01 22:05:07.275079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.275047    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.277603 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.277571    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:05:07.282952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.282923    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-702002-serving-cert\""
Nov 01 22:05:07.418161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.415160    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llts8\" (UniqueName: \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.418161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.415275    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.418161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.415514    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.418161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.415605    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.517166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.517083    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llts8\" (UniqueName: \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.517438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.517176    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.517438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.517370    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.517438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.517432    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.517988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.517968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.519852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.519820    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.520040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.520007    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.529534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.529508    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llts8\" (UniqueName: \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\") pod \"maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.594915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.594888    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:07.902991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.902959    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:05:07.905234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.905211    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:05:07.939001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:07.938970    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" event={"ID":"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373","Type":"ContainerStarted","Data":"712c8d13a929647a9363cc3ae3d3d53f356c5269bbc267f84209f274b6d64257"}
Nov 01 22:05:08.989914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.973065    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" event={"ID":"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373","Type":"ContainerStarted","Data":"6f42d489349d16459213e6970e94c0532aaa8675fc198760380c1e45d0c88a02"}
Nov 01 22:05:08.989914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.977102    5845 generic.go:356] "Generic (PLEG): container finished" podID="86da9fba-9c8e-49e1-8028-092beb13139b" containerID="56eb684e1be9c4dfcf8253cdac6b57c6d2b2b9dac3a16d7260ca60f16599ea29" exitCode=0
Nov 01 22:05:08.989914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.977190    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" event={"ID":"86da9fba-9c8e-49e1-8028-092beb13139b","Type":"ContainerDied","Data":"56eb684e1be9c4dfcf8253cdac6b57c6d2b2b9dac3a16d7260ca60f16599ea29"}
Nov 01 22:05:08.989914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.979166    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:08.992079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.992046    5845 generic.go:356] "Generic (PLEG): container finished" podID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerID="4b1b31a77dfb3296f07f0d419d7410f8dab671d24ef9fbb935df23dd48c84ad5" exitCode=0
Nov 01 22:05:08.992079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:08.992085    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" event={"ID":"9bb2beeb-625b-43f0-92d9-f7a55c3c1244","Type":"ContainerDied","Data":"4b1b31a77dfb3296f07f0d419d7410f8dab671d24ef9fbb935df23dd48c84ad5"}
Nov 01 22:05:09.002769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.002709    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" podStartSLOduration=1.394724182 podStartE2EDuration="2.002689761s" totalImagesPullingTime="607.965579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:09.000118183 +0000 UTC m=+1212.256297973" watchObservedRunningTime="2026-11-01 22:05:09.002689761 +0000 UTC m=+1212.258869528"
Nov 01 22:05:09.099043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.099016    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 22:05:09.172223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172192    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172273    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172371    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172421    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172500    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172605    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.172785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172615    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172682    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172723    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.172971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172836    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.172971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.172852    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\") pod \"86da9fba-9c8e-49e1-8028-092beb13139b\" (UID: \"86da9fba-9c8e-49e1-8028-092beb13139b\") "
Nov 01 22:05:09.173258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173137    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.173258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173173    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:09.173443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173407    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.173720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173700    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.173837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173724    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86da9fba-9c8e-49e1-8028-092beb13139b-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.173837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173739    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.173837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173752    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.173837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.173767    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.178672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.178640    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.180457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.179496    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:09.180457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.180028    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:05:09.180457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.180408    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx" pod "86da9fba-9c8e-49e1-8028-092beb13139b" (UID: "86da9fba-9c8e-49e1-8028-092beb13139b"). InnerVolumeSpecName "kube-api-access-nxpfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:09.180713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.180697    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 22:05:09.275168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275134    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275281    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275384    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275421    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275478    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275504    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275648    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275691    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275702    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.275852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275766    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.275852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275817    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\") pod \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\" (UID: \"9bb2beeb-625b-43f0-92d9-f7a55c3c1244\") "
Nov 01 22:05:09.275951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275907    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.275951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275918    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.276056 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.275959    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:09.276539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276519    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86da9fba-9c8e-49e1-8028-092beb13139b-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276539    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276548    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86da9fba-9c8e-49e1-8028-092beb13139b-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276558    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpfx\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-kube-api-access-nxpfx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276568    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276577    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276585    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276594    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.276693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.276602    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86da9fba-9c8e-49e1-8028-092beb13139b-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.277409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.277388    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:09.277658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.277642    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:05:09.278257 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.278243    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:09.278327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.278265    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299" pod "9bb2beeb-625b-43f0-92d9-f7a55c3c1244" (UID: "9bb2beeb-625b-43f0-92d9-f7a55c3c1244"). InnerVolumeSpecName "kube-api-access-nm299". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:09.377897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.377810    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.377897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.377839    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.377897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.377852    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.377897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.377864    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm299\" (UniqueName: \"kubernetes.io/projected/9bb2beeb-625b-43f0-92d9-f7a55c3c1244-kube-api-access-nm299\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:09.953699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.953316    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:05:09.954479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954451    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy"
Nov 01 22:05:09.954574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954480    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy"
Nov 01 22:05:09.954574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954528    5845 cpu_manager.go:414] "containerMap: removing container" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy"
Nov 01 22:05:09.954574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954536    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy"
Nov 01 22:05:09.954770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954753    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" containerName="istio-proxy"
Nov 01 22:05:09.954801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.954791    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" containerName="istio-proxy"
Nov 01 22:05:09.961021 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.960643    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:09.969279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.969250    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:05:09.978565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:09.978512    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3577a0-serving-cert\""
Nov 01 22:05:10.008404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.008362    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz" event={"ID":"86da9fba-9c8e-49e1-8028-092beb13139b","Type":"ContainerDied","Data":"3eb8142e9b5ef49684683037fb728a9b34218f1a95f2b632730363673654fe5a"}
Nov 01 22:05:10.008404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.008396    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"
Nov 01 22:05:10.008915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.008490    5845 scope.go:122] "RemoveContainer" containerID="56eb684e1be9c4dfcf8253cdac6b57c6d2b2b9dac3a16d7260ca60f16599ea29"
Nov 01 22:05:10.010787 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.010767    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"
Nov 01 22:05:10.010890 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.010781    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9" event={"ID":"9bb2beeb-625b-43f0-92d9-f7a55c3c1244","Type":"ContainerDied","Data":"200d1c91b20db4edf951a08d7d63a95d051a18b7fc347eca23c0bec51e40c201"}
Nov 01 22:05:10.024842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.024775    5845 scope.go:122] "RemoveContainer" containerID="4b1b31a77dfb3296f07f0d419d7410f8dab671d24ef9fbb935df23dd48c84ad5"
Nov 01 22:05:10.045016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.044688    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 22:05:10.049955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.049931    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-hqdjz"]
Nov 01 22:05:10.061058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.061036    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 22:05:10.067722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.067703    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-bgwd9"]
Nov 01 22:05:10.093157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.093133    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.093266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.093253    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.093371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.093351    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.093469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.093454    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qnwn\" (UniqueName: \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.194916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.194879    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.195087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.194942    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.195087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.195020    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qnwn\" (UniqueName: \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.195621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.195281    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.195903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.195881    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.198108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.198086    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.198237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.198174    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.207261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.207204    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qnwn\" (UniqueName: \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\") pod \"maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.301026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.300984    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:10.594442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.594416    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:05:10.595079 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:10.595049    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347a9163_6321_49db_a829_fc2b8cfa211e.slice/crio-d156e420213a760accf08b5a7369bb59a99c1222ad3ca6f6725ba19cee2590f6 WatchSource:0}: Error finding container d156e420213a760accf08b5a7369bb59a99c1222ad3ca6f6725ba19cee2590f6: Status 404 returned error can't find the container with id d156e420213a760accf08b5a7369bb59a99c1222ad3ca6f6725ba19cee2590f6
Nov 01 22:05:10.864930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.864851    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86da9fba-9c8e-49e1-8028-092beb13139b" path="/var/lib/kubelet/pods/86da9fba-9c8e-49e1-8028-092beb13139b/volumes"
Nov 01 22:05:10.865399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:10.865383    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb2beeb-625b-43f0-92d9-f7a55c3c1244" path="/var/lib/kubelet/pods/9bb2beeb-625b-43f0-92d9-f7a55c3c1244/volumes"
Nov 01 22:05:11.020564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:11.020530    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" event={"ID":"347a9163-6321-49db-a829-fc2b8cfa211e","Type":"ContainerStarted","Data":"d156e420213a760accf08b5a7369bb59a99c1222ad3ca6f6725ba19cee2590f6"}
Nov 01 22:05:12.038084 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:12.038043    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" event={"ID":"347a9163-6321-49db-a829-fc2b8cfa211e","Type":"ContainerStarted","Data":"84b06194d5774701dc959fff719909c28fce1f2779faa979435f6d3a71ca7958"}
Nov 01 22:05:12.066168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:12.066109    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" podStartSLOduration=2.492070553 podStartE2EDuration="3.066090533s" totalImagesPullingTime="574.01998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:12.060599732 +0000 UTC m=+1215.316779496" watchObservedRunningTime="2026-11-01 22:05:12.066090533 +0000 UTC m=+1215.322270297"
Nov 01 22:05:13.045596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.045570    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:13.393352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.393236    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:05:13.397784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.397757    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.409333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.409308    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:05:13.409459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.409409    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bb882c\""
Nov 01 22:05:13.493926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.493895    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.494190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.494173    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6frr\" (UniqueName: \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.596101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.595722    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6frr\" (UniqueName: \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.596101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.595881    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.597609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.597581    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.623954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.623868    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6frr\" (UniqueName: \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\") pod \"payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") " pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:13.723212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:13.723172    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:14.055489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:14.055455    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:05:14.055867 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:14.055669    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410fc61b_ffc4_4f28_a298_162b6c6c431c.slice/crio-0dc262b15eacc224a5040925dcc7a37dac1222bd45a6a290e5f2b367e498da6e WatchSource:0}: Error finding container 0dc262b15eacc224a5040925dcc7a37dac1222bd45a6a290e5f2b367e498da6e: Status 404 returned error can't find the container with id 0dc262b15eacc224a5040925dcc7a37dac1222bd45a6a290e5f2b367e498da6e
Nov 01 22:05:15.016714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:15.016687    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:05:15.068910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:15.068862    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" event={"ID":"410fc61b-ffc4-4f28-a298-162b6c6c431c","Type":"ContainerStarted","Data":"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"}
Nov 01 22:05:15.069433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:15.068937    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" event={"ID":"410fc61b-ffc4-4f28-a298-162b6c6c431c","Type":"ContainerStarted","Data":"0dc262b15eacc224a5040925dcc7a37dac1222bd45a6a290e5f2b367e498da6e"}
Nov 01 22:05:15.070065 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:15.070040    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:15.090453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:15.090396    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" podStartSLOduration=2.090377068 podStartE2EDuration="2.090377068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:15.087885455 +0000 UTC m=+1218.344065225" watchObservedRunningTime="2026-11-01 22:05:15.090377068 +0000 UTC m=+1218.346556830"
Nov 01 22:05:19.068692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:19.068660    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:05:21.973215 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:21.973168    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:05:22.013250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:22.013187    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:05:26.974215 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:26.974166    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" podUID="0866b18f-b24e-4169-984b-f01004ce6e3d" containerName="payload-processing" containerID="cri-o://3505ac835f3b789af092060d9ce5036da11c2d68f3b171132cda0e25155dd636" gracePeriod=25
Nov 01 22:05:27.013854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.013804    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" podUID="155bd7a0-52c5-4993-bb86-6654ff095133" containerName="payload-pre-processing" containerID="cri-o://203b0fd6c9d6115ed684b69f2f6bd462158eb07ea5f2c3fa3e9c0a70d863515e" gracePeriod=25
Nov 01 22:05:27.087635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.087611    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:05:27.241336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.241069    5845 generic.go:356] "Generic (PLEG): container finished" podID="0866b18f-b24e-4169-984b-f01004ce6e3d" containerID="3505ac835f3b789af092060d9ce5036da11c2d68f3b171132cda0e25155dd636" exitCode=0
Nov 01 22:05:27.241336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.241185    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" event={"ID":"0866b18f-b24e-4169-984b-f01004ce6e3d","Type":"ContainerDied","Data":"3505ac835f3b789af092060d9ce5036da11c2d68f3b171132cda0e25155dd636"}
Nov 01 22:05:27.243179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.243098    5845 generic.go:356] "Generic (PLEG): container finished" podID="155bd7a0-52c5-4993-bb86-6654ff095133" containerID="203b0fd6c9d6115ed684b69f2f6bd462158eb07ea5f2c3fa3e9c0a70d863515e" exitCode=0
Nov 01 22:05:27.243179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.243142    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" event={"ID":"155bd7a0-52c5-4993-bb86-6654ff095133","Type":"ContainerDied","Data":"203b0fd6c9d6115ed684b69f2f6bd462158eb07ea5f2c3fa3e9c0a70d863515e"}
Nov 01 22:05:27.341678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.341646    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:05:27.382511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.382485    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:05:27.506701 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.506621    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\") pod \"155bd7a0-52c5-4993-bb86-6654ff095133\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") "
Nov 01 22:05:27.506847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.506761    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\") pod \"0866b18f-b24e-4169-984b-f01004ce6e3d\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") "
Nov 01 22:05:27.506847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.506810    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\" (UniqueName: \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\") pod \"0866b18f-b24e-4169-984b-f01004ce6e3d\" (UID: \"0866b18f-b24e-4169-984b-f01004ce6e3d\") "
Nov 01 22:05:27.506847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.506840    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\" (UniqueName: \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\") pod \"155bd7a0-52c5-4993-bb86-6654ff095133\" (UID: \"155bd7a0-52c5-4993-bb86-6654ff095133\") "
Nov 01 22:05:27.507024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.506988    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume" pod "155bd7a0-52c5-4993-bb86-6654ff095133" (UID: "155bd7a0-52c5-4993-bb86-6654ff095133"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:27.507245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.507217    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume" pod "0866b18f-b24e-4169-984b-f01004ce6e3d" (UID: "0866b18f-b24e-4169-984b-f01004ce6e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:27.507509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.507493    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0866b18f-b24e-4169-984b-f01004ce6e3d-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:27.507550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.507511    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155bd7a0-52c5-4993-bb86-6654ff095133-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:27.508978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.508958    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw" pod "0866b18f-b24e-4169-984b-f01004ce6e3d" (UID: "0866b18f-b24e-4169-984b-f01004ce6e3d"). InnerVolumeSpecName "kube-api-access-cpmkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:27.509026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.508996    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd" pod "155bd7a0-52c5-4993-bb86-6654ff095133" (UID: "155bd7a0-52c5-4993-bb86-6654ff095133"). InnerVolumeSpecName "kube-api-access-g7zrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:27.609046 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.609015    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmkw\" (UniqueName: \"kubernetes.io/projected/0866b18f-b24e-4169-984b-f01004ce6e3d-kube-api-access-cpmkw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:27.609046 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:27.609045    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zrd\" (UniqueName: \"kubernetes.io/projected/155bd7a0-52c5-4993-bb86-6654ff095133-kube-api-access-g7zrd\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:28.251474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.251383    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"
Nov 01 22:05:28.251474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.251428    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx" event={"ID":"0866b18f-b24e-4169-984b-f01004ce6e3d","Type":"ContainerDied","Data":"99bc9ba5db5e08951234e0e6b23f682e16418c55e39516f6181fa097d6f374ac"}
Nov 01 22:05:28.252016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.251502    5845 scope.go:122] "RemoveContainer" containerID="3505ac835f3b789af092060d9ce5036da11c2d68f3b171132cda0e25155dd636"
Nov 01 22:05:28.253058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.253041    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"
Nov 01 22:05:28.253156 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.253066    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv" event={"ID":"155bd7a0-52c5-4993-bb86-6654ff095133","Type":"ContainerDied","Data":"de2513802195b8ac7a6aca7f020bc322c14f05dbcb783e5fcb0677cdbcd4ff27"}
Nov 01 22:05:28.262690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.262658    5845 scope.go:122] "RemoveContainer" containerID="203b0fd6c9d6115ed684b69f2f6bd462158eb07ea5f2c3fa3e9c0a70d863515e"
Nov 01 22:05:28.280205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.280176    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:05:28.290535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.290494    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a19582-85fd78cb9f-wf4xv"]
Nov 01 22:05:28.302521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.302466    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:05:28.308430 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.308398    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a19582-758fdc7b46-l7zwx"]
Nov 01 22:05:28.879993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.879955    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0866b18f-b24e-4169-984b-f01004ce6e3d" path="/var/lib/kubelet/pods/0866b18f-b24e-4169-984b-f01004ce6e3d/volumes"
Nov 01 22:05:28.881068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:28.881033    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155bd7a0-52c5-4993-bb86-6654ff095133" path="/var/lib/kubelet/pods/155bd7a0-52c5-4993-bb86-6654ff095133/volumes"
Nov 01 22:05:33.619323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:33.616510    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:05:33.619323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:33.616795    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy" containerID="cri-o://939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4" gracePeriod=30
Nov 01 22:05:35.130083 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:05:35.130044    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:05:36.043646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.043613    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.044453    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0866b18f-b24e-4169-984b-f01004ce6e3d" containerName="payload-processing"
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.046366    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0866b18f-b24e-4169-984b-f01004ce6e3d" containerName="payload-processing"
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.046416    5845 cpu_manager.go:414] "containerMap: removing container" podUID="155bd7a0-52c5-4993-bb86-6654ff095133" containerName="payload-pre-processing"
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.046426    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155bd7a0-52c5-4993-bb86-6654ff095133" containerName="payload-pre-processing"
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.046675    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0866b18f-b24e-4169-984b-f01004ce6e3d" containerName="payload-processing"
Nov 01 22:05:36.049004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.046689    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="155bd7a0-52c5-4993-bb86-6654ff095133" containerName="payload-pre-processing"
Nov 01 22:05:36.075549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.075521    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.088256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.088222    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:05:36.090230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.090203    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-de0cba\""
Nov 01 22:05:36.090585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.090539    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-de0cba-dockercfg-g6kwr\""
Nov 01 22:05:36.103525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.103497    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:05:36.110412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.110383    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgq9s\" (UniqueName: \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.110600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.110585    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.114436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.114410    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.134625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.134438    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:05:36.233319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.229639    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgq9s\" (UniqueName: \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.233319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.229734    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.233319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.230658    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.349374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.340041    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnmt\" (UniqueName: \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.349374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.340205    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.448322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.447458    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgq9s\" (UniqueName: \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\") pod \"payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.476319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.464680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnmt\" (UniqueName: \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.500599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.500562    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.501390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.501362    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.505058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.505031    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnmt\" (UniqueName: \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\") pod \"payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:36.711937 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.711855    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:36.760321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:36.757592    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:37.038736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.038561    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:05:37.041467 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:37.041103    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d002b5_67d0_4d80_a9c5_4b6f72d14837.slice/crio-08589b67f74436893c44c99889177ea1ae756ba070f792d9058d70c4e914bce3 WatchSource:0}: Error finding container 08589b67f74436893c44c99889177ea1ae756ba070f792d9058d70c4e914bce3: Status 404 returned error can't find the container with id 08589b67f74436893c44c99889177ea1ae756ba070f792d9058d70c4e914bce3
Nov 01 22:05:37.100451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.100423    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:05:37.100955 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:37.100928    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468b5cf4_14a0_4954_8c5b_870be68ecf1e.slice/crio-ede993efc5dcb8388ba84eb831087f5f5194c4a62ec36368167c06824016bafb WatchSource:0}: Error finding container ede993efc5dcb8388ba84eb831087f5f5194c4a62ec36368167c06824016bafb: Status 404 returned error can't find the container with id ede993efc5dcb8388ba84eb831087f5f5194c4a62ec36368167c06824016bafb
Nov 01 22:05:37.515547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.515507    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" event={"ID":"468b5cf4-14a0-4954-8c5b-870be68ecf1e","Type":"ContainerStarted","Data":"a8143bbff9469610b961b0ebe2c0281df234b4caaa8a970d314b4b458cfa3446"}
Nov 01 22:05:37.516003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.515729    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:37.516003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.515772    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" event={"ID":"468b5cf4-14a0-4954-8c5b-870be68ecf1e","Type":"ContainerStarted","Data":"ede993efc5dcb8388ba84eb831087f5f5194c4a62ec36368167c06824016bafb"}
Nov 01 22:05:37.517351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.517326    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" event={"ID":"71d002b5-67d0-4d80-a9c5-4b6f72d14837","Type":"ContainerStarted","Data":"656f91fa997998cf54bab9346f8035a57b639539af77e9708d8566be8751e025"}
Nov 01 22:05:37.517476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.517373    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" event={"ID":"71d002b5-67d0-4d80-a9c5-4b6f72d14837","Type":"ContainerStarted","Data":"08589b67f74436893c44c99889177ea1ae756ba070f792d9058d70c4e914bce3"}
Nov 01 22:05:37.517476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.517419    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:37.566901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.566841    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podStartSLOduration=1.566821323 podStartE2EDuration="1.566821323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:37.560711402 +0000 UTC m=+1240.816891166" watchObservedRunningTime="2026-11-01 22:05:37.566821323 +0000 UTC m=+1240.823001087"
Nov 01 22:05:37.627824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:37.627763    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podStartSLOduration=1.627745219 podStartE2EDuration="1.627745219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:37.622719954 +0000 UTC m=+1240.878899729" watchObservedRunningTime="2026-11-01 22:05:37.627745219 +0000 UTC m=+1240.883924983"
Nov 01 22:05:39.058313 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.055858    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:05:39.170565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170480    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.170565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170550    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170583    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170631    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170767    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170807    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170877    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.170924    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171040    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\") pod \"0388a06b-168c-42e5-93b6-8b2776675272\" (UID: \"0388a06b-168c-42e5-93b6-8b2776675272\") "
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171411    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171428    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171608    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171655    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.171803    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.172617    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.172670    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.172692    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388a06b-168c-42e5-93b6-8b2776675272-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.172708    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.172840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.172723    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.174368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.174108    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:39.174368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.174169    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:39.174571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.174544    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:05:39.174712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.174693    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm" pod "0388a06b-168c-42e5-93b6-8b2776675272" (UID: "0388a06b-168c-42e5-93b6-8b2776675272"). InnerVolumeSpecName "kube-api-access-qljzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:39.274931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.274890    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388a06b-168c-42e5-93b6-8b2776675272-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.274931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.274928    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.275136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.274942    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388a06b-168c-42e5-93b6-8b2776675272-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.275136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.274957    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljzm\" (UniqueName: \"kubernetes.io/projected/0388a06b-168c-42e5-93b6-8b2776675272-kube-api-access-qljzm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:39.548109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.548000    5845 generic.go:356] "Generic (PLEG): container finished" podID="0388a06b-168c-42e5-93b6-8b2776675272" containerID="939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4" exitCode=0
Nov 01 22:05:39.548368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.548271    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" event={"ID":"0388a06b-168c-42e5-93b6-8b2776675272","Type":"ContainerDied","Data":"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"}
Nov 01 22:05:39.548448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.548377    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb" event={"ID":"0388a06b-168c-42e5-93b6-8b2776675272","Type":"ContainerDied","Data":"fdb86a281b606fa5742e197d434db73834fa06b503ea2e6ef08d22494f71c0ca"}
Nov 01 22:05:39.548448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.548432    5845 scope.go:122] "RemoveContainer" containerID="939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"
Nov 01 22:05:39.549503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.549291    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"
Nov 01 22:05:39.579824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.579800    5845 scope.go:122] "RemoveContainer" containerID="939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"
Nov 01 22:05:39.580252 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:05:39.580232    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4\": container with ID starting with 939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4 not found: ID does not exist" containerID="939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"
Nov 01 22:05:39.580336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.580261    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4"} err="failed to get container status \"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4\": rpc error: code = NotFound desc = could not find container \"939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4\": container with ID starting with 939c4b8bb4de341bc89f52e5d6e147371d59ac0c31e24325e7d3c3922c6db6a4 not found: ID does not exist"
Nov 01 22:05:39.595643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.595612    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:05:39.602120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:39.602088    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-m2hqb"]
Nov 01 22:05:40.061475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:40.061435    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:05:40.061873 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:40.061492    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:05:40.865775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:40.865743    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0388a06b-168c-42e5-93b6-8b2776675272" path="/var/lib/kubelet/pods/0388a06b-168c-42e5-93b6-8b2776675272/volumes"
Nov 01 22:05:42.596670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.596633    5845 generic.go:356] "Generic (PLEG): container finished" podID="cd205146-f5cd-4603-aa2a-cf5384811873" containerID="65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325" exitCode=0
Nov 01 22:05:42.597122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.596709    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerDied","Data":"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"}
Nov 01 22:05:42.597198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.597182    5845 scope.go:122] "RemoveContainer" containerID="65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"
Nov 01 22:05:42.599279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.599249    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:42.599422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.599338    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:42.599422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:42.599374    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:43.606443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:43.606408    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerStarted","Data":"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"}
Nov 01 22:05:44.276322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:44.275682    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:05:44.277205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:44.277177    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:44.277473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:44.277438    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:45.276457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:45.276415    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:45.277077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:45.277044    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:46.275794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:46.275750    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:46.276001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:46.275809    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:47.273320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:47.272734    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:47.273320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:47.272838    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:48.271062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.270751    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:48.271062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.270828    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:48.558379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.550567    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:05:48.580358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.570647    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:05:48.755352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.755287    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 22:05:48.755750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.755727    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 22:05:48.756402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.756363    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy" containerID="cri-o://14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9" gracePeriod=30
Nov 01 22:05:48.756821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:48.756794    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy" containerID="cri-o://6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894" gracePeriod=30
Nov 01 22:05:49.275324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:49.271852    5845 patch_prober.go:29] interesting pod/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 22:05:49.275324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:49.271952    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 22:05:50.298656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:50.298618    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:05:50.301002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:50.300433    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:05:50.308214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:50.308179    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:05:53.758505 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.758470    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:05:53.759276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.759258    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy"
Nov 01 22:05:53.759363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.759277    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy"
Nov 01 22:05:53.759428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.759422    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0388a06b-168c-42e5-93b6-8b2776675272" containerName="istio-proxy"
Nov 01 22:05:53.763416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.763387    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:53.775338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.775286    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:05:53.777554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.777528    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18657f-dockercfg-l22m7\""
Nov 01 22:05:53.823838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.823799    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18657f\""
Nov 01 22:05:53.935544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.935494    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fbw\" (UniqueName: \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:53.945608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:53.936552    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.133935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.125677    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fbw\" (UniqueName: \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.133935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.126215    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.152596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.152556    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.155170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.155118    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fbw\" (UniqueName: \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\") pod \"payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.325549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.325519    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 22:05:54.337217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.337191    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 22:05:54.387406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.385131    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.458419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.458365    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.458532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.458458    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.458600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.458531    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.458652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.458611    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.462258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.462202    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.467135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467105    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.467274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467153    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.467385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467278    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.467385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467362    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.467507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467412    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.467507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467501    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.467614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467544    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.467614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467581    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.467722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467656    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467817    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467877    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467896    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.467906    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468012    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\") pod \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\" (UID: \"30cc4be8-1e86-447d-b363-da9a4a2c3c6b\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468039    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\") pod \"23715e0a-24d9-4d1f-91c5-a02cf518c781\" (UID: \"23715e0a-24d9-4d1f-91c5-a02cf518c781\") "
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468070    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468111    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468313    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468407    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468721    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.468806    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469049    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.469243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469149    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469418    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469670    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469736    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469870    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469890    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23715e0a-24d9-4d1f-91c5-a02cf518c781-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469905    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469919    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469933    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469947    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469961    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469974    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.469991    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.470004    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.470018    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.470720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.470033    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.473256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.473228    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:05:54.473448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.473411    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:54.473600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.473562    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l" pod "30cc4be8-1e86-447d-b363-da9a4a2c3c6b" (UID: "30cc4be8-1e86-447d-b363-da9a4a2c3c6b"). InnerVolumeSpecName "kube-api-access-8sl5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:54.474882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.474850    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "kube-api-access-zpt97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:05:54.475752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.475724    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo" pod "23715e0a-24d9-4d1f-91c5-a02cf518c781" (UID: "23715e0a-24d9-4d1f-91c5-a02cf518c781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:05:54.572118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572075    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sl5l\" (UniqueName: \"kubernetes.io/projected/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-kube-api-access-8sl5l\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.572118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572110    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpt97\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-kube-api-access-zpt97\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.572118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572120    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.572118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572128    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.572410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572138    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cc4be8-1e86-447d-b363-da9a4a2c3c6b-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.572410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.572148    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23715e0a-24d9-4d1f-91c5-a02cf518c781-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:05:54.693521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.692674    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:05:54.888343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.888291    5845 generic.go:356] "Generic (PLEG): container finished" podID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerID="14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9" exitCode=0
Nov 01 22:05:54.888343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.888329    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" event={"ID":"23715e0a-24d9-4d1f-91c5-a02cf518c781","Type":"ContainerDied","Data":"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"}
Nov 01 22:05:54.888835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.888354    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"
Nov 01 22:05:54.888835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.888388    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx" event={"ID":"23715e0a-24d9-4d1f-91c5-a02cf518c781","Type":"ContainerDied","Data":"9fad217a949bf95d68ee953e71460333cdfd76eda5309c5570a4c970896f0800"}
Nov 01 22:05:54.888835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.888421    5845 scope.go:122] "RemoveContainer" containerID="14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"
Nov 01 22:05:54.891278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.890876    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" event={"ID":"b11f2650-ec48-4e3f-97b9-29f63c78695f","Type":"ContainerStarted","Data":"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"}
Nov 01 22:05:54.891278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.890937    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" event={"ID":"b11f2650-ec48-4e3f-97b9-29f63c78695f","Type":"ContainerStarted","Data":"146bd0c692f45ec692b601fc4b2bba4fe60752b95c28b6e11bd8f109e03cfda2"}
Nov 01 22:05:54.891278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.890972    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:05:54.892823 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.892801    5845 generic.go:356] "Generic (PLEG): container finished" podID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerID="6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894" exitCode=0
Nov 01 22:05:54.892952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.892838    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"
Nov 01 22:05:54.892952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.892880    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" event={"ID":"30cc4be8-1e86-447d-b363-da9a4a2c3c6b","Type":"ContainerDied","Data":"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"}
Nov 01 22:05:54.892952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.892933    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg" event={"ID":"30cc4be8-1e86-447d-b363-da9a4a2c3c6b","Type":"ContainerDied","Data":"f0204fb144afadb879eaddea13d7416ac651e285bddb8feef2d7a3397be4fe40"}
Nov 01 22:05:54.901218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.901196    5845 scope.go:122] "RemoveContainer" containerID="14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"
Nov 01 22:05:54.901549 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:05:54.901527    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9\": container with ID starting with 14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9 not found: ID does not exist" containerID="14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"
Nov 01 22:05:54.901595 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.901558    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9"} err="failed to get container status \"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9\": rpc error: code = NotFound desc = could not find container \"14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9\": container with ID starting with 14fb60b27650995f84226181316b5d78c1dd6d358d57fbaad215fedc3e1311e9 not found: ID does not exist"
Nov 01 22:05:54.901595 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.901576    5845 scope.go:122] "RemoveContainer" containerID="6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"
Nov 01 22:05:54.912763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.912738    5845 scope.go:122] "RemoveContainer" containerID="6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"
Nov 01 22:05:54.913179 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:05:54.913148    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894\": container with ID starting with 6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894 not found: ID does not exist" containerID="6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"
Nov 01 22:05:54.913277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.913190    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894"} err="failed to get container status \"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894\": rpc error: code = NotFound desc = could not find container \"6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894\": container with ID starting with 6f3dbdfd7e79369b0380b4acb1f27049b342c83f093d419d9abafb487b033894 not found: ID does not exist"
Nov 01 22:05:54.914768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.914724    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" podStartSLOduration=1.914711346 podStartE2EDuration="1.914711346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:54.91234764 +0000 UTC m=+1258.168527401" watchObservedRunningTime="2026-11-01 22:05:54.914711346 +0000 UTC m=+1258.170891133"
Nov 01 22:05:54.941139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.941037    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 22:05:54.946486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.946454    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-2tzsx"]
Nov 01 22:05:54.961468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.961434    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 22:05:54.967255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:54.967230    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-zzffg"]
Nov 01 22:05:55.438233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.438191    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:05:55.439229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439196    5845 cpu_manager.go:414] "containerMap: removing container" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy"
Nov 01 22:05:55.439229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439220    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy"
Nov 01 22:05:55.439229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439233    5845 cpu_manager.go:414] "containerMap: removing container" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy"
Nov 01 22:05:55.439229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439238    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy"
Nov 01 22:05:55.439611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439392    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" containerName="istio-proxy"
Nov 01 22:05:55.439611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.439404    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" containerName="istio-proxy"
Nov 01 22:05:55.445441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.445409    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.454920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.454891    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-21dd16de-openshift-default-dockercfg-8mdj7\""
Nov 01 22:05:55.458087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.457316    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:05:55.462979 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.462951    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.493579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.474469    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:05:55.501723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.501688    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:05:55.598118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598082    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598132    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598239    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598265    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598333    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598469    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598519    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598579    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598636 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598613    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8n7\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598638    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598719    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzld\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598800    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598820    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.598903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598884    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.598959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598948    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.599016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598970    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.599016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.598984    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.599085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.599061    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.700813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.700723    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.700813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.700798    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.700856    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.700892    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.700941    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701004    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701039    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701066    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701103    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8n7\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701135    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701194    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzld\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701290    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701334    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701369    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701471    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.701533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701503    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701527    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.701600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.701561    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.702025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.702001    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.702263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.702244    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.702909 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.702883    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.703886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.703859    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.704801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.704781    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.704916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.704831    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.705090 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.705045    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.705180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.705089    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.705326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.705272    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.705406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.705364    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.708637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.708401    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.708637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.708590    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.708637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.708606    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.709325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.709290    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.714708 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.714674    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.717051 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.717024    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzld\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:55.717417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.717395    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.717557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.717535    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8n7\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\") pod \"e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") " pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.830339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.819179    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:55.830339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:55.819786    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:56.267552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.267523    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:05:56.267962 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:56.267935    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb573949_71f5_40e3_b059_ad5031b672b7.slice/crio-ae0a78e8ea6ba4a4c4b39a4fd4b7788b3bb414892a62422b64d50f0e9752d52b WatchSource:0}: Error finding container ae0a78e8ea6ba4a4c4b39a4fd4b7788b3bb414892a62422b64d50f0e9752d52b: Status 404 returned error can't find the container with id ae0a78e8ea6ba4a4c4b39a4fd4b7788b3bb414892a62422b64d50f0e9752d52b
Nov 01 22:05:56.271112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.271075    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.271209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.271179    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.271266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.271224    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.350161 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:05:56.350127    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205b36e2_d185_41c3_900c_65840e221b3c.slice/crio-e3c478ed7372232db63e08228f915abc1e93ef29cb50dd372522ee4ba87ecb53 WatchSource:0}: Error finding container e3c478ed7372232db63e08228f915abc1e93ef29cb50dd372522ee4ba87ecb53: Status 404 returned error can't find the container with id e3c478ed7372232db63e08228f915abc1e93ef29cb50dd372522ee4ba87ecb53
Nov 01 22:05:56.350503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.350477    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:05:56.353530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.353493    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.353628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.353582    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.353679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.353628    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:05:56.869145 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.869108    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23715e0a-24d9-4d1f-91c5-a02cf518c781" path="/var/lib/kubelet/pods/23715e0a-24d9-4d1f-91c5-a02cf518c781/volumes"
Nov 01 22:05:56.869852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:56.869832    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30cc4be8-1e86-447d-b363-da9a4a2c3c6b" path="/var/lib/kubelet/pods/30cc4be8-1e86-447d-b363-da9a4a2c3c6b/volumes"
Nov 01 22:05:57.080332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.080247    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" event={"ID":"205b36e2-d185-41c3-900c-65840e221b3c","Type":"ContainerStarted","Data":"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"}
Nov 01 22:05:57.080332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.080335    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" event={"ID":"205b36e2-d185-41c3-900c-65840e221b3c","Type":"ContainerStarted","Data":"e3c478ed7372232db63e08228f915abc1e93ef29cb50dd372522ee4ba87ecb53"}
Nov 01 22:05:57.083199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.083161    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" event={"ID":"cb573949-71f5-40e3-b059-ad5031b672b7","Type":"ContainerStarted","Data":"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"}
Nov 01 22:05:57.083487 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.083461    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" event={"ID":"cb573949-71f5-40e3-b059-ad5031b672b7","Type":"ContainerStarted","Data":"ae0a78e8ea6ba4a4c4b39a4fd4b7788b3bb414892a62422b64d50f0e9752d52b"}
Nov 01 22:05:57.126381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.126258    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podStartSLOduration=2.126243987 podStartE2EDuration="2.126243987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:57.123526334 +0000 UTC m=+1260.379706096" watchObservedRunningTime="2026-11-01 22:05:57.126243987 +0000 UTC m=+1260.382423750"
Nov 01 22:05:57.822966 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.821591    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:05:57.829313 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.824426    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:05:57.829313 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.824491    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:05:57.839801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.839772    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:05:57.840770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.840745    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:05:57.840956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:57.840784    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:05:58.823382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:58.821756    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:05:58.823382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:58.821803    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:05:58.839874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:58.839828    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:05:58.840015 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:58.839897    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:05:59.822471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.822390    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:05:59.822471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.822445    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:05:59.839782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.839753    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:05:59.840132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.839801    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:05:59.973322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.969939    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podStartSLOduration=4.969916361 podStartE2EDuration="4.969916361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:05:57.172671837 +0000 UTC m=+1260.428851601" watchObservedRunningTime="2026-11-01 22:05:59.969916361 +0000 UTC m=+1263.226096126"
Nov 01 22:05:59.997320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.985437    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:05:59.997320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:05:59.985685    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" podUID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" containerName="maas-api" containerID="cri-o://5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5" gracePeriod=30
Nov 01 22:06:00.823390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.817487    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:06:00.823574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.823475    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:06:00.823574 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.823525    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:06:00.851374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.845394    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:06:00.851374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.845452    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:06:00.952356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.948766    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\" (UniqueName: \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\") pod \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") "
Nov 01 22:06:00.952356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.949026    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\") pod \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") "
Nov 01 22:06:00.952356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.949108    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\") pod \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") "
Nov 01 22:06:00.952356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:00.949201    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\") pod \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\" (UID: \"08562f88-02b9-4aab-b4bd-0ad93aaa83d6\") "
Nov 01 22:06:01.016714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.016676    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca" pod "08562f88-02b9-4aab-b4bd-0ad93aaa83d6" (UID: "08562f88-02b9-4aab-b4bd-0ad93aaa83d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:01.018342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.018281    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls" pod "08562f88-02b9-4aab-b4bd-0ad93aaa83d6" (UID: "08562f88-02b9-4aab-b4bd-0ad93aaa83d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:06:01.018500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.018473    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs" pod "08562f88-02b9-4aab-b4bd-0ad93aaa83d6" (UID: "08562f88-02b9-4aab-b4bd-0ad93aaa83d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:06:01.018629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.018610    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf" pod "08562f88-02b9-4aab-b4bd-0ad93aaa83d6" (UID: "08562f88-02b9-4aab-b4bd-0ad93aaa83d6"). InnerVolumeSpecName "kube-api-access-nxvtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:01.069321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.068744    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvtf\" (UniqueName: \"kubernetes.io/projected/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-kube-api-access-nxvtf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:01.069321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.068782    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:01.069321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.068798    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:01.069321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.068812    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08562f88-02b9-4aab-b4bd-0ad93aaa83d6-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:01.141566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.141470    5845 generic.go:356] "Generic (PLEG): container finished" podID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" containerID="5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5" exitCode=0
Nov 01 22:06:01.141566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.141528    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"
Nov 01 22:06:01.141788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.141561    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" event={"ID":"08562f88-02b9-4aab-b4bd-0ad93aaa83d6","Type":"ContainerDied","Data":"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"}
Nov 01 22:06:01.141788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.141616    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx" event={"ID":"08562f88-02b9-4aab-b4bd-0ad93aaa83d6","Type":"ContainerDied","Data":"f2da6781f27a15eef89da0dfce1223766c337ab71272ceaf11d75b4025eec4c9"}
Nov 01 22:06:01.141788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.141644    5845 scope.go:122] "RemoveContainer" containerID="5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"
Nov 01 22:06:01.205281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.205249    5845 scope.go:122] "RemoveContainer" containerID="5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"
Nov 01 22:06:01.206346 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:01.206246    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5\": container with ID starting with 5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5 not found: ID does not exist" containerID="5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"
Nov 01 22:06:01.206346 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.206311    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5"} err="failed to get container status \"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5\": rpc error: code = NotFound desc = could not find container \"5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5\": container with ID starting with 5c799178fd8e064ebe7efd5847b7cbcc3477ce3a99ceaf73c2bb14d00d8f47c5 not found: ID does not exist"
Nov 01 22:06:01.225736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.225704    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:06:01.229389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.229363    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a19582-6487545c6c-m49fx"]
Nov 01 22:06:01.822324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.821857    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:06:01.822324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.821931    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:06:01.839591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.839561    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:06:01.839743 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:01.839602    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:06:02.821844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:02.821806    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 01 22:06:02.822256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:02.821868    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 01 22:06:02.840376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:02.840341    5845 patch_prober.go:29] interesting pod/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 22:06:02.840559 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:02.840399    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 22:06:02.867071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:02.867028    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" path="/var/lib/kubelet/pods/08562f88-02b9-4aab-b4bd-0ad93aaa83d6/volumes"
Nov 01 22:06:03.827608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.827575    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:06:03.828006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.827921    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:06:03.828654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.828633    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:06:03.850663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.850636    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:06:03.850998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.850974    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:06:03.851610 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:03.851593    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:06:04.826314 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:04.826261    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:06:06.010521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:06.010484    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:06:06.560501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:06.560467    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:06:06.578693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:06.578663    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:06:10.180971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.180891    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:06:10.181754 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.181735    5845 cpu_manager.go:414] "containerMap: removing container" podUID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" containerName="maas-api"
Nov 01 22:06:10.181799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.181756    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" containerName="maas-api"
Nov 01 22:06:10.181876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.181867    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="08562f88-02b9-4aab-b4bd-0ad93aaa83d6" containerName="maas-api"
Nov 01 22:06:10.187045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.187028    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.193094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.193071    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-18657f-serving-cert\""
Nov 01 22:06:10.201539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.197521    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:06:10.308030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.307998    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.308225 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.308200    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjkn\" (UniqueName: \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.308415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.308395    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.308476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.308433    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.410289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.410246    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.410537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.410516    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjkn\" (UniqueName: \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.410789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.410765    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.410897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.410821    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.411870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.411838    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.414204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.414180    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.418281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.418255    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.422936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.422913    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjkn\" (UniqueName: \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\") pod \"maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.508349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.507376    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:10.810561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:10.810529    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:06:10.811701 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:10.811672    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b66b0b_e7e1_4d42_b1fd_c39dc6f7b143.slice/crio-3a9e3890bb94b64c79c7aa1eb1958ccb13d4cf9bb60146836cea147f2230e281 WatchSource:0}: Error finding container 3a9e3890bb94b64c79c7aa1eb1958ccb13d4cf9bb60146836cea147f2230e281: Status 404 returned error can't find the container with id 3a9e3890bb94b64c79c7aa1eb1958ccb13d4cf9bb60146836cea147f2230e281
Nov 01 22:06:11.253748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:11.253714    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" event={"ID":"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143","Type":"ContainerStarted","Data":"3a9e3890bb94b64c79c7aa1eb1958ccb13d4cf9bb60146836cea147f2230e281"}
Nov 01 22:06:11.566372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:11.561369    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" podUID="96ecbada-df58-4383-89ef-5d3cf69a0f65" containerName="payload-processing" containerID="cri-o://a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c" gracePeriod=25
Nov 01 22:06:11.601011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:11.580523    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" podUID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" containerName="payload-pre-processing" containerID="cri-o://ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add" gracePeriod=25
Nov 01 22:06:12.039548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.039520    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:06:12.048026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.048002    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:06:12.055752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.055729    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\") pod \"96ecbada-df58-4383-89ef-5d3cf69a0f65\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") "
Nov 01 22:06:12.055877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.055846    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\" (UniqueName: \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\") pod \"96ecbada-df58-4383-89ef-5d3cf69a0f65\" (UID: \"96ecbada-df58-4383-89ef-5d3cf69a0f65\") "
Nov 01 22:06:12.056094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.056071    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume" pod "96ecbada-df58-4383-89ef-5d3cf69a0f65" (UID: "96ecbada-df58-4383-89ef-5d3cf69a0f65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:12.056462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.056444    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96ecbada-df58-4383-89ef-5d3cf69a0f65-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:12.057855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.057828    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk" pod "96ecbada-df58-4383-89ef-5d3cf69a0f65" (UID: "96ecbada-df58-4383-89ef-5d3cf69a0f65"). InnerVolumeSpecName "kube-api-access-7g5mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:12.157576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.157501    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\" (UniqueName: \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\") pod \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") "
Nov 01 22:06:12.157746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.157733    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\") pod \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\" (UID: \"d5ecf857-b357-42b1-a7be-57f7bfbbdd08\") "
Nov 01 22:06:12.158115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.158090    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume" pod "d5ecf857-b357-42b1-a7be-57f7bfbbdd08" (UID: "d5ecf857-b357-42b1-a7be-57f7bfbbdd08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:12.158437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.158415    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g5mk\" (UniqueName: \"kubernetes.io/projected/96ecbada-df58-4383-89ef-5d3cf69a0f65-kube-api-access-7g5mk\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:12.158517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.158440    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:12.159585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.159564    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x" pod "d5ecf857-b357-42b1-a7be-57f7bfbbdd08" (UID: "d5ecf857-b357-42b1-a7be-57f7bfbbdd08"). InnerVolumeSpecName "kube-api-access-55p4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:12.259954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.259921    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55p4x\" (UniqueName: \"kubernetes.io/projected/d5ecf857-b357-42b1-a7be-57f7bfbbdd08-kube-api-access-55p4x\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:12.262813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.262789    5845 generic.go:356] "Generic (PLEG): container finished" podID="96ecbada-df58-4383-89ef-5d3cf69a0f65" containerID="a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c" exitCode=0
Nov 01 22:06:12.262938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.262834    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"
Nov 01 22:06:12.262938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.262872    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" event={"ID":"96ecbada-df58-4383-89ef-5d3cf69a0f65","Type":"ContainerDied","Data":"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"}
Nov 01 22:06:12.262938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.262923    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j" event={"ID":"96ecbada-df58-4383-89ef-5d3cf69a0f65","Type":"ContainerDied","Data":"e06e5d996eb6935f9f2005f68e6cf7308b686ff8dac4f436efb02401f957fc72"}
Nov 01 22:06:12.263069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.262946    5845 scope.go:122] "RemoveContainer" containerID="a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"
Nov 01 22:06:12.264667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.264639    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" event={"ID":"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143","Type":"ContainerStarted","Data":"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"}
Nov 01 22:06:12.264775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.264701    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:12.266101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.266075    5845 generic.go:356] "Generic (PLEG): container finished" podID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" containerID="ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add" exitCode=0
Nov 01 22:06:12.266208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.266104    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" event={"ID":"d5ecf857-b357-42b1-a7be-57f7bfbbdd08","Type":"ContainerDied","Data":"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"}
Nov 01 22:06:12.266208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.266128    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"
Nov 01 22:06:12.266208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.266149    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h" event={"ID":"d5ecf857-b357-42b1-a7be-57f7bfbbdd08","Type":"ContainerDied","Data":"99ce1f57dad44dd54f8552d972ace0dd0a62938eb5fdb42e6012d168847daee9"}
Nov 01 22:06:12.276450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.276362    5845 scope.go:122] "RemoveContainer" containerID="a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"
Nov 01 22:06:12.276777 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:12.276756    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c\": container with ID starting with a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c not found: ID does not exist" containerID="a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"
Nov 01 22:06:12.276860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.276785    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c"} err="failed to get container status \"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c\": rpc error: code = NotFound desc = could not find container \"a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c\": container with ID starting with a175b6746641d8e1d6789f436efc5ea236307605802286eee37b8aad0fbd571c not found: ID does not exist"
Nov 01 22:06:12.276860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.276802    5845 scope.go:122] "RemoveContainer" containerID="ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"
Nov 01 22:06:12.288470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.288453    5845 scope.go:122] "RemoveContainer" containerID="ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"
Nov 01 22:06:12.288746 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:12.288718    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add\": container with ID starting with ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add not found: ID does not exist" containerID="ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"
Nov 01 22:06:12.288844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.288752    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add"} err="failed to get container status \"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add\": rpc error: code = NotFound desc = could not find container \"ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add\": container with ID starting with ce79445a5dd00d4de8e9c2d3b50e70324d11ac401e69fcaa6407bccc37160add not found: ID does not exist"
Nov 01 22:06:12.293639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.293602    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" podStartSLOduration=1.708284027 podStartE2EDuration="2.293590192s" totalImagesPullingTime="585.306165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:12.292518806 +0000 UTC m=+1275.548698569" watchObservedRunningTime="2026-11-01 22:06:12.293590192 +0000 UTC m=+1275.549769956"
Nov 01 22:06:12.319859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.319830    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:06:12.324073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.324053    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-004180-845f96955-lw94h"]
Nov 01 22:06:12.333023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.332960    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:06:12.336919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.336899    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-004180-548d486474-4m87j"]
Nov 01 22:06:12.863445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.863411    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ecbada-df58-4383-89ef-5d3cf69a0f65" path="/var/lib/kubelet/pods/96ecbada-df58-4383-89ef-5d3cf69a0f65/volumes"
Nov 01 22:06:12.863954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:12.863938    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" path="/var/lib/kubelet/pods/d5ecf857-b357-42b1-a7be-57f7bfbbdd08/volumes"
Nov 01 22:06:13.356819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:13.356785    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:06:13.357364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:13.357036    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy" containerID="cri-o://f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91" gracePeriod=30
Nov 01 22:06:13.373052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:13.373017    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:06:13.373479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:13.373447    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy" containerID="cri-o://4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957" gracePeriod=30
Nov 01 22:06:17.191978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.191938    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:06:17.192717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192701    5845 cpu_manager.go:414] "containerMap: removing container" podUID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" containerName="payload-pre-processing"
Nov 01 22:06:17.192782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192718    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" containerName="payload-pre-processing"
Nov 01 22:06:17.192782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192727    5845 cpu_manager.go:414] "containerMap: removing container" podUID="96ecbada-df58-4383-89ef-5d3cf69a0f65" containerName="payload-processing"
Nov 01 22:06:17.192782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192733    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ecbada-df58-4383-89ef-5d3cf69a0f65" containerName="payload-processing"
Nov 01 22:06:17.192907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192897    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ecf857-b357-42b1-a7be-57f7bfbbdd08" containerName="payload-pre-processing"
Nov 01 22:06:17.192942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.192909    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ecbada-df58-4383-89ef-5d3cf69a0f65" containerName="payload-processing"
Nov 01 22:06:17.196679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.196658    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.205411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.205381    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d0a5de\""
Nov 01 22:06:17.207245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.207217    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d0a5de-dockercfg-gl8jk\""
Nov 01 22:06:17.210789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.210762    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:06:17.269323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.268354    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfh5\" (UniqueName: \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.269323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.268523    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.269564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.269334    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:06:17.284352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.280515    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:06:17.284352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.280613    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.369891 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.369852    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfh5\" (UniqueName: \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.370162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.370029    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.370162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.370065    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.370162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.370096    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6fz\" (UniqueName: \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.371063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.371028    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.389364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.389326    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfh5\" (UniqueName: \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\") pod \"payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.483415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.473210    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.483415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.473261    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6fz\" (UniqueName: \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.483415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.474353    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.488288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.488253    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6fz\" (UniqueName: \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\") pod \"payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.527188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.527150    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:17.604705 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.603134    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:17.821896 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:17.821869    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1428094a_6382_4d52_8786_da737f78c348.slice/crio-1f6340c3c5a7182d33e5e6dac8f8b599f4cd209188f9e8becf278b5942b5c443 WatchSource:0}: Error finding container 1f6340c3c5a7182d33e5e6dac8f8b599f4cd209188f9e8becf278b5942b5c443: Status 404 returned error can't find the container with id 1f6340c3c5a7182d33e5e6dac8f8b599f4cd209188f9e8becf278b5942b5c443
Nov 01 22:06:17.822104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.822080    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:06:17.937179 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:17.937152    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8285bc_132c_40be_924d_c15debf27b9d.slice/crio-959c387d7b0d732732ccba1dd5898a91e5289e8b60b3c5c169a0b65650175b46 WatchSource:0}: Error finding container 959c387d7b0d732732ccba1dd5898a91e5289e8b60b3c5c169a0b65650175b46: Status 404 returned error can't find the container with id 959c387d7b0d732732ccba1dd5898a91e5289e8b60b3c5c169a0b65650175b46
Nov 01 22:06:17.937349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:17.937326    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:06:18.282603 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.282573    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:06:18.364366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.364330    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" event={"ID":"0b8285bc-132c-40be-924d-c15debf27b9d","Type":"ContainerStarted","Data":"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"}
Nov 01 22:06:18.364547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.364400    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:18.364547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.364423    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" event={"ID":"0b8285bc-132c-40be-924d-c15debf27b9d","Type":"ContainerStarted","Data":"959c387d7b0d732732ccba1dd5898a91e5289e8b60b3c5c169a0b65650175b46"}
Nov 01 22:06:18.366773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.366741    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" event={"ID":"1428094a-6382-4d52-8786-da737f78c348","Type":"ContainerStarted","Data":"8eceb5124b86806a0d35bbaccc52a96ed37d30d085dd09bee571290e4613edd7"}
Nov 01 22:06:18.366913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.366787    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" event={"ID":"1428094a-6382-4d52-8786-da737f78c348","Type":"ContainerStarted","Data":"1f6340c3c5a7182d33e5e6dac8f8b599f4cd209188f9e8becf278b5942b5c443"}
Nov 01 22:06:18.366913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.366836    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:18.385498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.385440    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" podStartSLOduration=1.385420688 podStartE2EDuration="1.385420688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:18.382658631 +0000 UTC m=+1281.638838395" watchObservedRunningTime="2026-11-01 22:06:18.385420688 +0000 UTC m=+1281.641600451"
Nov 01 22:06:18.401454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.401399    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" podStartSLOduration=1.401381614 podStartE2EDuration="1.401381614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:18.400245669 +0000 UTC m=+1281.656425430" watchObservedRunningTime="2026-11-01 22:06:18.401381614 +0000 UTC m=+1281.657561384"
Nov 01 22:06:18.720526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.720494    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 22:06:18.720747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.720723    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy" containerID="cri-o://8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a" gracePeriod=30
Nov 01 22:06:18.727096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.727067    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:06:18.765110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.765082    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:06:18.802877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.802832    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803121    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.803361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803340    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803428    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803504    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803596    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803641    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803743 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803727    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803791    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.803881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803806    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.803935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.803920    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\") pod \"205b36e2-d185-41c3-900c-65840e221b3c\" (UID: \"205b36e2-d185-41c3-900c-65840e221b3c\") "
Nov 01 22:06:18.804718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.804695    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.805041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805018    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.805330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805292    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:18.805494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805475    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.805857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805833    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.805857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805857    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.805984 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.805872    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.823337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.817130    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:06:18.823337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.817451    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.823337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.817516    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:18.823337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.821625    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld" pod "205b36e2-d185-41c3-900c-65840e221b3c" (UID: "205b36e2-d185-41c3-900c-65840e221b3c"). InnerVolumeSpecName "kube-api-access-rlzld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:18.907025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.906994    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907045    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907176    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907237    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907263    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.907436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907362    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907411    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907494    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907612    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.907826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907635    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.907826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907693    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\") pod \"cb573949-71f5-40e3-b059-ad5031b672b7\" (UID: \"cb573949-71f5-40e3-b059-ad5031b672b7\") "
Nov 01 22:06:18.908339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908256    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.908480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:18.908587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.907988    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:18.908916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908875    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.908916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908909    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908926    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908941    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908954    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205b36e2-d185-41c3-900c-65840e221b3c-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908968    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908982    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzld\" (UniqueName: \"kubernetes.io/projected/205b36e2-d185-41c3-900c-65840e221b3c-kube-api-access-rlzld\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.908996    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205b36e2-d185-41c3-900c-65840e221b3c-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.909010    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205b36e2-d185-41c3-900c-65840e221b3c-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.909023    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb573949-71f5-40e3-b059-ad5031b672b7-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:18.909910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.909888    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:18.910196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.910166    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "kube-api-access-hf8n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:18.910366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.910350    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:06:18.910431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:18.910421    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy" pod "cb573949-71f5-40e3-b059-ad5031b672b7" (UID: "cb573949-71f5-40e3-b059-ad5031b672b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:19.010968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.010937    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb573949-71f5-40e3-b059-ad5031b672b7-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:19.010968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.010966    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:19.011137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.010976    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf8n7\" (UniqueName: \"kubernetes.io/projected/cb573949-71f5-40e3-b059-ad5031b672b7-kube-api-access-hf8n7\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:19.011137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.010985    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb573949-71f5-40e3-b059-ad5031b672b7-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:19.374926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.374896    5845 generic.go:356] "Generic (PLEG): container finished" podID="205b36e2-d185-41c3-900c-65840e221b3c" containerID="4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957" exitCode=0
Nov 01 22:06:19.375407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.374946    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"
Nov 01 22:06:19.375407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.374984    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" event={"ID":"205b36e2-d185-41c3-900c-65840e221b3c","Type":"ContainerDied","Data":"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"}
Nov 01 22:06:19.375407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.375040    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg" event={"ID":"205b36e2-d185-41c3-900c-65840e221b3c","Type":"ContainerDied","Data":"e3c478ed7372232db63e08228f915abc1e93ef29cb50dd372522ee4ba87ecb53"}
Nov 01 22:06:19.375407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.375069    5845 scope.go:122] "RemoveContainer" containerID="4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"
Nov 01 22:06:19.376710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.376688    5845 generic.go:356] "Generic (PLEG): container finished" podID="cb573949-71f5-40e3-b059-ad5031b672b7" containerID="f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91" exitCode=0
Nov 01 22:06:19.376841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.376756    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"
Nov 01 22:06:19.376841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.376799    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" event={"ID":"cb573949-71f5-40e3-b059-ad5031b672b7","Type":"ContainerDied","Data":"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"}
Nov 01 22:06:19.376943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.376851    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp" event={"ID":"cb573949-71f5-40e3-b059-ad5031b672b7","Type":"ContainerDied","Data":"ae0a78e8ea6ba4a4c4b39a4fd4b7788b3bb414892a62422b64d50f0e9752d52b"}
Nov 01 22:06:19.387352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.387334    5845 scope.go:122] "RemoveContainer" containerID="4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"
Nov 01 22:06:19.387632 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:19.387610    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957\": container with ID starting with 4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957 not found: ID does not exist" containerID="4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"
Nov 01 22:06:19.387689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.387639    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957"} err="failed to get container status \"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957\": rpc error: code = NotFound desc = could not find container \"4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957\": container with ID starting with 4857f1b7bafe4d6914fcad2cee66f718a1395d086902266560e57d691a9c3957 not found: ID does not exist"
Nov 01 22:06:19.387689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.387654    5845 scope.go:122] "RemoveContainer" containerID="f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"
Nov 01 22:06:19.403646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.403620    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:06:19.409547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.409527    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-tpbcg"]
Nov 01 22:06:19.414177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.414160    5845 scope.go:122] "RemoveContainer" containerID="f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"
Nov 01 22:06:19.414490 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:19.414471    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91\": container with ID starting with f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91 not found: ID does not exist" containerID="f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"
Nov 01 22:06:19.414543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.414497    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91"} err="failed to get container status \"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91\": rpc error: code = NotFound desc = could not find container \"f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91\": container with ID starting with f13d428bee822cb7482db31860b179fdf2bb3abc515f38ab10f2c8d310bcdf91 not found: ID does not exist"
Nov 01 22:06:19.426611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.426589    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:06:19.429984 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:19.429966    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-21dd16de-openshift-default-5db99d957-xm2fp"]
Nov 01 22:06:20.863218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:20.863187    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205b36e2-d185-41c3-900c-65840e221b3c" path="/var/lib/kubelet/pods/205b36e2-d185-41c3-900c-65840e221b3c/volumes"
Nov 01 22:06:20.863974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:20.863950    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" path="/var/lib/kubelet/pods/cb573949-71f5-40e3-b059-ad5031b672b7/volumes"
Nov 01 22:06:24.012403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.012375    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 22:06:24.140719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140684    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.140907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140730    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.140907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140800    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.140907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140847    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.140907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140879    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.141134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140912    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.141134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140936    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.141134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.140996    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:24.141134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141004    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.141134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141122    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\") pod \"beae6af7-38cc-4619-a61f-fe4eb01608d0\" (UID: \"beae6af7-38cc-4619-a61f-fe4eb01608d0\") "
Nov 01 22:06:24.141480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141113    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:24.141538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141327    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:24.141538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141282    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:24.141538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.141448    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:24.142105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.142085    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.142202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.142110    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beae6af7-38cc-4619-a61f-fe4eb01608d0-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.142202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.142124    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.142202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.142136    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.142202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.142149    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.143565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.143538    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:06:24.143565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.143546    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:06:24.144448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.144425    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:24.144529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.144452    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6" pod "beae6af7-38cc-4619-a61f-fe4eb01608d0" (UID: "beae6af7-38cc-4619-a61f-fe4eb01608d0"). InnerVolumeSpecName "kube-api-access-cp4x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:24.244399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.244318    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.244399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.244351    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.244399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.244380    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.244399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.244397    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4x6\" (UniqueName: \"kubernetes.io/projected/beae6af7-38cc-4619-a61f-fe4eb01608d0-kube-api-access-cp4x6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:24.425019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.424978    5845 generic.go:356] "Generic (PLEG): container finished" podID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerID="8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a" exitCode=0
Nov 01 22:06:24.425186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.425032    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"
Nov 01 22:06:24.425186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.425046    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" event={"ID":"beae6af7-38cc-4619-a61f-fe4eb01608d0","Type":"ContainerDied","Data":"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"}
Nov 01 22:06:24.425186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.425123    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5" event={"ID":"beae6af7-38cc-4619-a61f-fe4eb01608d0","Type":"ContainerDied","Data":"247781f5ff9e2e5cb5a4989d32d1caf5fdf541bf91b65b78d490d71dc5e8852c"}
Nov 01 22:06:24.425186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.425155    5845 scope.go:122] "RemoveContainer" containerID="8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"
Nov 01 22:06:24.449998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.448449    5845 scope.go:122] "RemoveContainer" containerID="8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"
Nov 01 22:06:24.449998 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:24.448829    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a\": container with ID starting with 8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a not found: ID does not exist" containerID="8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"
Nov 01 22:06:24.449998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.448858    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a"} err="failed to get container status \"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a\": rpc error: code = NotFound desc = could not find container \"8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a\": container with ID starting with 8cdc736aa492a35c0d75a118e3eb661d75efa5fd264b64c3a529d6e7761ce53a not found: ID does not exist"
Nov 01 22:06:24.467910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.467884    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 22:06:24.473472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.471769    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-q95p5"]
Nov 01 22:06:24.863088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:24.863055    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" path="/var/lib/kubelet/pods/beae6af7-38cc-4619-a61f-fe4eb01608d0/volumes"
Nov 01 22:06:29.385329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:29.384682    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:06:29.385329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:29.384797    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:06:34.828597 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:34.828548    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:06:42.163406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.163373    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:06:42.164687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164659    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy"
Nov 01 22:06:42.164687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164686    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy"
Nov 01 22:06:42.164865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164764    5845 cpu_manager.go:414] "containerMap: removing container" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy"
Nov 01 22:06:42.164865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164772    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy"
Nov 01 22:06:42.164865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164799    5845 cpu_manager.go:414] "containerMap: removing container" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy"
Nov 01 22:06:42.164865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164808    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy"
Nov 01 22:06:42.165053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.164991    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb573949-71f5-40e3-b059-ad5031b672b7" containerName="istio-proxy"
Nov 01 22:06:42.165053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.165023    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="beae6af7-38cc-4619-a61f-fe4eb01608d0" containerName="istio-proxy"
Nov 01 22:06:42.165053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.165033    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="205b36e2-d185-41c3-900c-65840e221b3c" containerName="istio-proxy"
Nov 01 22:06:42.170434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.170413    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.183588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.183552    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:06:42.219546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.219513    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3577a0-openshift-default-dockercfg-nmmqq\""
Nov 01 22:06:42.260282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260251    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260366    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxqj\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260391    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260510    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260544    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260763    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260797    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260896    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.260945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.260933    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362223    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362262    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362337    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362363    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362385    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362410    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxqj\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362563    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362646    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362707    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362748    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.362904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.362780    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.363054 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.363031    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.363340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.363318    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.364797 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.364777    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.364926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.364905    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.374107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.374087    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.374466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.374445    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxqj\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\") pod \"e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") " pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.494375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.494347    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:42.819374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.819329    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:06:42.821065 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:42.821035    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56283c_80a8_4cf3_a77a_0757a7bde40c.slice/crio-c2212a4f7ffa755634e33fd3119345010726794d3d4973551d24d60c47940dfc WatchSource:0}: Error finding container c2212a4f7ffa755634e33fd3119345010726794d3d4973551d24d60c47940dfc: Status 404 returned error can't find the container with id c2212a4f7ffa755634e33fd3119345010726794d3d4973551d24d60c47940dfc
Nov 01 22:06:42.824387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.824353    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:42.824503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.824420    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:42.824503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:42.824448    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:43.680684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:43.680648    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" event={"ID":"ee56283c-80a8-4cf3-a77a-0757a7bde40c","Type":"ContainerStarted","Data":"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"}
Nov 01 22:06:43.681129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:43.680713    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" event={"ID":"ee56283c-80a8-4cf3-a77a-0757a7bde40c","Type":"ContainerStarted","Data":"c2212a4f7ffa755634e33fd3119345010726794d3d4973551d24d60c47940dfc"}
Nov 01 22:06:43.708886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:43.708839    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podStartSLOduration=1.70882524 podStartE2EDuration="1.70882524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:43.70417035 +0000 UTC m=+1306.960350125" watchObservedRunningTime="2026-11-01 22:06:43.70882524 +0000 UTC m=+1306.965005002"
Nov 01 22:06:44.308701 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:44.308668    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:06:44.495350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:44.495316    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:44.496577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:44.496550    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:44.496698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:44.496611    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:45.495477 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:45.495436    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:45.495897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:45.495505    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:46.337440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.337403    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:06:46.341897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.341878    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.377224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.377170    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17957f\""
Nov 01 22:06:46.378079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.378052    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-17957f-dockercfg-l6gx2\""
Nov 01 22:06:46.382571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.382548    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:06:46.432365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.432286    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9ts\" (UniqueName: \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.432503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.432449    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.517507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.514216    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:46.517507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.514279    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:46.548320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.547491    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9ts\" (UniqueName: \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.548320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.547597    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.552316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.548742    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.577320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.576035    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9ts\" (UniqueName: \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\") pod \"payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.692369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.691627    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:46.823424 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.823387    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:06:46.884692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.882645    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.894648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.894618    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-edd32a-openshift-default-dockercfg-lkssj\""
Nov 01 22:06:46.898988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.898966    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.980580    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.980648    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.980728    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.980769    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpdg\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.980923    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.981005    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.981045    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.981233    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:46.987314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:46.981416    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083072    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083129    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083153    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083236    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083352    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083397    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083427    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083490    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083527    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpdg\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083602    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083655    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083804    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.083980 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.083941    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.084120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.084093    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.085945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.085921    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.086061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.086038    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.103466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.103440    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:06:47.103908 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:47.103887    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3378d78_f15c_4d7b_9efe_352664f6a03b.slice/crio-ece199ceb2e48b674ca695cefd2cf6266842b971c4d53ac0d02e20a27bd9a2cf WatchSource:0}: Error finding container ece199ceb2e48b674ca695cefd2cf6266842b971c4d53ac0d02e20a27bd9a2cf: Status 404 returned error can't find the container with id ece199ceb2e48b674ca695cefd2cf6266842b971c4d53ac0d02e20a27bd9a2cf
Nov 01 22:06:47.104683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.104657    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.104824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.104768    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpdg\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.205242 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.205216    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:47.495200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.495167    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:47.495402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.495229    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:47.521352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.521321    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:06:47.521743 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:06:47.521586    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4683d5_2dca_4f63_a9ab_2bd314e8051d.slice/crio-125622bcd3acb42d5d593b97398a807c3a056705f66303a2cec0c62b237478b9 WatchSource:0}: Error finding container 125622bcd3acb42d5d593b97398a807c3a056705f66303a2cec0c62b237478b9: Status 404 returned error can't find the container with id 125622bcd3acb42d5d593b97398a807c3a056705f66303a2cec0c62b237478b9
Nov 01 22:06:47.524410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.524381    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:47.524513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.524439    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:47.524513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.524467    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:06:47.796164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.796064    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" event={"ID":"ca4683d5-2dca-4f63-a9ab-2bd314e8051d","Type":"ContainerStarted","Data":"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"}
Nov 01 22:06:47.796455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.796418    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" event={"ID":"ca4683d5-2dca-4f63-a9ab-2bd314e8051d","Type":"ContainerStarted","Data":"125622bcd3acb42d5d593b97398a807c3a056705f66303a2cec0c62b237478b9"}
Nov 01 22:06:47.799920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.799190    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" event={"ID":"c3378d78-f15c-4d7b-9efe-352664f6a03b","Type":"ContainerStarted","Data":"cfd00ffddf63c3e70931f717e5c4b4bdcd840af9a04819c6e3c672b5e2a56e09"}
Nov 01 22:06:47.799920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.799242    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" event={"ID":"c3378d78-f15c-4d7b-9efe-352664f6a03b","Type":"ContainerStarted","Data":"ece199ceb2e48b674ca695cefd2cf6266842b971c4d53ac0d02e20a27bd9a2cf"}
Nov 01 22:06:47.799920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.799497    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:06:47.826438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.825366    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podStartSLOduration=1.8253474729999999 podStartE2EDuration="1.825347473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:47.821792349 +0000 UTC m=+1311.077972114" watchObservedRunningTime="2026-11-01 22:06:47.825347473 +0000 UTC m=+1311.081527238"
Nov 01 22:06:47.849973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:47.849277    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" podStartSLOduration=1.8492594709999999 podStartE2EDuration="1.849259471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:06:47.840412808 +0000 UTC m=+1311.096592571" watchObservedRunningTime="2026-11-01 22:06:47.849259471 +0000 UTC m=+1311.105439234"
Nov 01 22:06:48.206078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:48.205995    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:48.207630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:48.207604    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:48.207761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:48.207660    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:48.498285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:48.497802    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:48.498285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:48.497872    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:49.205630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.205594    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:49.206060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.205655    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:49.309189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.309143    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" podUID="410fc61b-ffc4-4f28-a298-162b6c6c431c" containerName="payload-processing" containerID="cri-o://123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92" gracePeriod=25
Nov 01 22:06:49.495554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.495518    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 01 22:06:49.495729 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.495568    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 01 22:06:49.693871 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.693841    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:06:49.733715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.733674    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\") pod \"410fc61b-ffc4-4f28-a298-162b6c6c431c\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") "
Nov 01 22:06:49.733898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.733878    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\" (UniqueName: \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\") pod \"410fc61b-ffc4-4f28-a298-162b6c6c431c\" (UID: \"410fc61b-ffc4-4f28-a298-162b6c6c431c\") "
Nov 01 22:06:49.734146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.734119    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume" pod "410fc61b-ffc4-4f28-a298-162b6c6c431c" (UID: "410fc61b-ffc4-4f28-a298-162b6c6c431c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:06:49.734717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.734689    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/410fc61b-ffc4-4f28-a298-162b6c6c431c-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:49.742364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.736991    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr" pod "410fc61b-ffc4-4f28-a298-162b6c6c431c" (UID: "410fc61b-ffc4-4f28-a298-162b6c6c431c"). InnerVolumeSpecName "kube-api-access-q6frr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:06:49.826118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.826084    5845 generic.go:356] "Generic (PLEG): container finished" podID="410fc61b-ffc4-4f28-a298-162b6c6c431c" containerID="123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92" exitCode=0
Nov 01 22:06:49.827275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.827251    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"
Nov 01 22:06:49.829564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.828527    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" event={"ID":"410fc61b-ffc4-4f28-a298-162b6c6c431c","Type":"ContainerDied","Data":"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"}
Nov 01 22:06:49.829564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.828590    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj" event={"ID":"410fc61b-ffc4-4f28-a298-162b6c6c431c","Type":"ContainerDied","Data":"0dc262b15eacc224a5040925dcc7a37dac1222bd45a6a290e5f2b367e498da6e"}
Nov 01 22:06:49.829564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.828621    5845 scope.go:122] "RemoveContainer" containerID="123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"
Nov 01 22:06:49.837513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.837485    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6frr\" (UniqueName: \"kubernetes.io/projected/410fc61b-ffc4-4f28-a298-162b6c6c431c-kube-api-access-q6frr\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:06:49.882351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.880480    5845 scope.go:122] "RemoveContainer" containerID="123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"
Nov 01 22:06:49.882351 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:06:49.880977    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92\": container with ID starting with 123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92 not found: ID does not exist" containerID="123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"
Nov 01 22:06:49.882351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.881017    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92"} err="failed to get container status \"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92\": rpc error: code = NotFound desc = could not find container \"123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92\": container with ID starting with 123ca4fe565e55152416c8e9d5e48e605484a4ea97d9faeb47ef908aab6c6b92 not found: ID does not exist"
Nov 01 22:06:49.882952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.882904    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:06:49.886940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:49.886918    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bb882c-8486dc96fb-llqsj"]
Nov 01 22:06:50.206620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.206537    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:50.206620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.206597    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:50.500991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.500954    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:50.501632 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.501613    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:50.502246 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.502225    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:06:50.888416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:50.888335    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="410fc61b-ffc4-4f28-a298-162b6c6c431c" path="/var/lib/kubelet/pods/410fc61b-ffc4-4f28-a298-162b6c6c431c/volumes"
Nov 01 22:06:51.207113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:51.206547    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:51.207113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:51.206620    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:52.206446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:52.206370    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:52.206446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:52.206432    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:53.205788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:53.205751    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:53.206172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:53.205804    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:54.206110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:54.206074    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 01 22:06:54.206561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:54.206120    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 01 22:06:55.211754 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:55.211710    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:55.212522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:55.212275    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:55.213091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:55.213070    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:06:58.815810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:06:58.815781    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:07:02.868338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.868266    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:07:02.869492 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.869470    5845 cpu_manager.go:414] "containerMap: removing container" podUID="410fc61b-ffc4-4f28-a298-162b6c6c431c" containerName="payload-processing"
Nov 01 22:07:02.869575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.869496    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="410fc61b-ffc4-4f28-a298-162b6c6c431c" containerName="payload-processing"
Nov 01 22:07:02.869739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.869725    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="410fc61b-ffc4-4f28-a298-162b6c6c431c" containerName="payload-processing"
Nov 01 22:07:02.876366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.876342    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:02.877240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.877214    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:07:02.883723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.883700    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-17957f-serving-cert\""
Nov 01 22:07:02.905608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.905579    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:02.906123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.906004    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:02.906123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.906099    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:02.906345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:02.906203    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8dp\" (UniqueName: \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.008095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.008033    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.008287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.008169    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.008287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.008244    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8dp\" (UniqueName: \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.008429 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:03.008332    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-serving-cert: secret "maas-api-e2e-inf-a-17957f-serving-cert" not found
Nov 01 22:07:03.008429 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:03.008413    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls podName:9e568845-e244-4484-a952-876893d3f477 nodeName:}" failed. No retries permitted until 2026-11-01 22:07:03.508388084 +0000 UTC m=+1326.764567841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls") pod "maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" (UID: "9e568845-e244-4484-a952-876893d3f477") : secret "maas-api-e2e-inf-a-17957f-serving-cert" not found
Nov 01 22:07:03.008664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.008631    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.009007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.008984    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.047539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.047502    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.050789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.050757    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8dp\" (UniqueName: \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.517115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.517079    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.519496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.519471    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") pod \"maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:03.793691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:03.793594    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:04.131910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:04.131881    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:07:04.132293 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:04.132268    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e568845_e244_4484_a952_876893d3f477.slice/crio-808918f0ed289ce8e030d648f9818eea5bf716f7343b86e828260ddd6df15539 WatchSource:0}: Error finding container 808918f0ed289ce8e030d648f9818eea5bf716f7343b86e828260ddd6df15539: Status 404 returned error can't find the container with id 808918f0ed289ce8e030d648f9818eea5bf716f7343b86e828260ddd6df15539
Nov 01 22:07:04.821427 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:04.821365    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:07:04.993602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:04.993565    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" event={"ID":"9e568845-e244-4484-a952-876893d3f477","Type":"ContainerStarted","Data":"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"}
Nov 01 22:07:04.993793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:04.993618    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" event={"ID":"9e568845-e244-4484-a952-876893d3f477","Type":"ContainerStarted","Data":"808918f0ed289ce8e030d648f9818eea5bf716f7343b86e828260ddd6df15539"}
Nov 01 22:07:04.993793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:04.993767    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:05.017877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.017820    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" podStartSLOduration=2.483835309 podStartE2EDuration="3.01780819s" totalImagesPullingTime="533.972881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:05.013554513 +0000 UTC m=+1328.269734277" watchObservedRunningTime="2026-11-01 22:07:05.01780819 +0000 UTC m=+1328.273987952"
Nov 01 22:07:05.514391 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.514351    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:07:05.519706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.519676    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.528306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.528265    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:07:05.549900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.549875    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.550110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.550094    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5gd\" (UniqueName: \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.614936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.614912    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9d8b5d-dockercfg-vz66r\""
Nov 01 22:07:05.615800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.615785    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9d8b5d\""
Nov 01 22:07:05.678598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.678549    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5gd\" (UniqueName: \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.678857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.678833    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.680571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.680531    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.703132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.703102    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5gd\" (UniqueName: \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\") pod \"payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:05.847311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:05.846528    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:06.138908 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:06.138879    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9f93d5_7660_4fd1_8340_fca5525292dd.slice/crio-8005befb9d7705b44dcf5cce5d98b462245342562a4274a250dca5ebe2fa2453 WatchSource:0}: Error finding container 8005befb9d7705b44dcf5cce5d98b462245342562a4274a250dca5ebe2fa2453: Status 404 returned error can't find the container with id 8005befb9d7705b44dcf5cce5d98b462245342562a4274a250dca5ebe2fa2453
Nov 01 22:07:06.139206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:06.139182    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:07:07.016358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:07.016318    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" event={"ID":"bf9f93d5-7660-4fd1-8340-fca5525292dd","Type":"ContainerStarted","Data":"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"}
Nov 01 22:07:07.016744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:07.016497    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:07.016744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:07.016539    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" event={"ID":"bf9f93d5-7660-4fd1-8340-fca5525292dd","Type":"ContainerStarted","Data":"8005befb9d7705b44dcf5cce5d98b462245342562a4274a250dca5ebe2fa2453"}
Nov 01 22:07:07.039085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:07.039040    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" podStartSLOduration=2.039027253 podStartE2EDuration="2.039027253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:07.035179095 +0000 UTC m=+1330.291358860" watchObservedRunningTime="2026-11-01 22:07:07.039027253 +0000 UTC m=+1330.295207048"
Nov 01 22:07:10.153317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:10.139793    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:07:10.153317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:10.140047    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy" containerID="cri-o://a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab" gracePeriod=30
Nov 01 22:07:11.022338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:11.022169    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:07:14.259950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.259913    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:07:14.276612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.276137    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:07:14.276612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.276252    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.323217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.321626    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0f9b7e\""
Nov 01 22:07:14.353452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.351947    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0f9b7e-dockercfg-9mt99\""
Nov 01 22:07:14.445138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.444658    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bc9n\" (UniqueName: \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.445138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.444720    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.546224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.546134    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bc9n\" (UniqueName: \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.546224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.546179    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.546994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.546973    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.563342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.559958    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bc9n\" (UniqueName: \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\") pod \"payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.652875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.652845    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:14.938529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:14.938501    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:07:14.938715 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:14.938692    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d552b5b_7b02_467c_9f79_6d53a1125cb5.slice/crio-492fec0c7cd607bfae07a51bfb571d393214dc5933927d9f4398d9b83948497c WatchSource:0}: Error finding container 492fec0c7cd607bfae07a51bfb571d393214dc5933927d9f4398d9b83948497c: Status 404 returned error can't find the container with id 492fec0c7cd607bfae07a51bfb571d393214dc5933927d9f4398d9b83948497c
Nov 01 22:07:15.096910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.096875    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" event={"ID":"1d552b5b-7b02-467c-9f79-6d53a1125cb5","Type":"ContainerStarted","Data":"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"}
Nov 01 22:07:15.097105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.096936    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" event={"ID":"1d552b5b-7b02-467c-9f79-6d53a1125cb5","Type":"ContainerStarted","Data":"492fec0c7cd607bfae07a51bfb571d393214dc5933927d9f4398d9b83948497c"}
Nov 01 22:07:15.097105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.097078    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:15.116536 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.116492    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" podStartSLOduration=1.116480377 podStartE2EDuration="1.116480377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:15.114620365 +0000 UTC m=+1338.370800141" watchObservedRunningTime="2026-11-01 22:07:15.116480377 +0000 UTC m=+1338.372660139"
Nov 01 22:07:15.429125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.429091    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:07:15.469395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469363    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469404    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469436    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469585    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469647    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469687    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469691    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:07:15.469919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469768    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.469919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469832    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:07:15.469919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469841    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.470075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.469933    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\") pod \"cd205146-f5cd-4603-aa2a-cf5384811873\" (UID: \"cd205146-f5cd-4603-aa2a-cf5384811873\") "
Nov 01 22:07:15.470380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470338    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:07:15.470525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470382    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:07:15.470763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470733    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.470763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470750    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:07:15.470763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470762    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.470920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470777    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.470920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.470790    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.473484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.473126    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:07:15.473484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.473358    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:07:15.473753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.473733    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "kube-api-access-szk7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:07:15.474219 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.474193    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token" pod "cd205146-f5cd-4603-aa2a-cf5384811873" (UID: "cd205146-f5cd-4603-aa2a-cf5384811873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:07:15.571844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.571818    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk7c\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-kube-api-access-szk7c\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.571906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.571847    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd205146-f5cd-4603-aa2a-cf5384811873-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.571906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.571858    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd205146-f5cd-4603-aa2a-cf5384811873-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.571906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.571867    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd205146-f5cd-4603-aa2a-cf5384811873-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:15.571906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:15.571875    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd205146-f5cd-4603-aa2a-cf5384811873-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:16.109395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.109348    5845 generic.go:356] "Generic (PLEG): container finished" podID="cd205146-f5cd-4603-aa2a-cf5384811873" containerID="a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab" exitCode=0
Nov 01 22:07:16.109625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.109578    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerDied","Data":"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"}
Nov 01 22:07:16.109715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.109665    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7" event={"ID":"cd205146-f5cd-4603-aa2a-cf5384811873","Type":"ContainerDied","Data":"710c058c8ffdcf1aab7c477aacdd22323257706329a68497b0963c4a594f0690"}
Nov 01 22:07:16.109715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.109696    5845 scope.go:122] "RemoveContainer" containerID="a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"
Nov 01 22:07:16.110058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.109558    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"
Nov 01 22:07:16.126541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.126516    5845 scope.go:122] "RemoveContainer" containerID="65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"
Nov 01 22:07:16.143167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.143141    5845 scope.go:122] "RemoveContainer" containerID="a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"
Nov 01 22:07:16.145785 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:16.145752    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab\": container with ID starting with a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab not found: ID does not exist" containerID="a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"
Nov 01 22:07:16.146088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.146054    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab"} err="failed to get container status \"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab\": rpc error: code = NotFound desc = could not find container \"a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab\": container with ID starting with a13744c0c5a8c3a97b366953e0169125a72067c03b0fe7b0477e38a8f8faabab not found: ID does not exist"
Nov 01 22:07:16.146214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.146199    5845 scope.go:122] "RemoveContainer" containerID="65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"
Nov 01 22:07:16.146436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.146260    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:07:16.146836 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:16.146809    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325\": container with ID starting with 65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325 not found: ID does not exist" containerID="65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"
Nov 01 22:07:16.146931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.146842    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325"} err="failed to get container status \"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325\": rpc error: code = NotFound desc = could not find container \"65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325\": container with ID starting with 65ae28108efe51566ea044e8ef7220b45e028c87ffeab0155a569574d4832325 not found: ID does not exist"
Nov 01 22:07:16.150825 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.150802    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0390e1-openshift-default-5499d6bdcf-v25t7"]
Nov 01 22:07:16.642669 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.642636    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:07:16.643471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643452    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.643471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643470    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.643584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643481    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.643584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643486    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.643651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643639    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.643651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.643648    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" containerName="istio-proxy"
Nov 01 22:07:16.649147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.649125    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.657577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.657535    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f2760d\""
Nov 01 22:07:16.657577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.657556    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f2760d-dockercfg-t8q6k\""
Nov 01 22:07:16.658131 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.658107    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:07:16.688945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.688915    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7b2b\" (UniqueName: \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.689085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.688957    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.794320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.792592    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7b2b\" (UniqueName: \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.794320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.792649    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.796323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.795267    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.842045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.836080    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7b2b\" (UniqueName: \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\") pod \"payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:16.871046 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.871012    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd205146-f5cd-4603-aa2a-cf5384811873" path="/var/lib/kubelet/pods/cd205146-f5cd-4603-aa2a-cf5384811873/volumes"
Nov 01 22:07:16.966730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:16.966699    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:17.290824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:17.290796    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:07:17.291460 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:17.291430    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5fdb64_a54c_4184_900a_136a92af01ed.slice/crio-6271ad90f26716e10dae3d90579e7095d47ef93ec80e1aaa2e1556b40323d75a WatchSource:0}: Error finding container 6271ad90f26716e10dae3d90579e7095d47ef93ec80e1aaa2e1556b40323d75a: Status 404 returned error can't find the container with id 6271ad90f26716e10dae3d90579e7095d47ef93ec80e1aaa2e1556b40323d75a
Nov 01 22:07:18.035137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:18.035106    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:07:18.155482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:18.155445    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" event={"ID":"3a5fdb64-a54c-4184-900a-136a92af01ed","Type":"ContainerStarted","Data":"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"}
Nov 01 22:07:18.155628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:18.155528    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" event={"ID":"3a5fdb64-a54c-4184-900a-136a92af01ed","Type":"ContainerStarted","Data":"6271ad90f26716e10dae3d90579e7095d47ef93ec80e1aaa2e1556b40323d75a"}
Nov 01 22:07:18.155628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:18.155555    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:18.178011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:18.177962    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" podStartSLOduration=2.177947084 podStartE2EDuration="2.177947084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:18.173091985 +0000 UTC m=+1341.429271739" watchObservedRunningTime="2026-11-01 22:07:18.177947084 +0000 UTC m=+1341.434126872"
Nov 01 22:07:26.117576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:26.117534    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:07:29.167936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:29.167901    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:07:31.846127 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.846095    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:07:31.868020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.866981    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.878217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.874640    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:07:31.971045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971005    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.971253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971183    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.971392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971371    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.971463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971414    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.971534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971499    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.972149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971675    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.972149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971780    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.972149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971852    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmws6\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:31.972149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:31.971987    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090544    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090608    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090713    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090744    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmws6\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090902    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.090981    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.091068    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.091316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.091178    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.114364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.091968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.114364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.093753    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.114364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.096671    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.114364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.096899    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.114364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.097519    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.136994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.136960    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.139795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.139769    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.141766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.141743    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmws6\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.142254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.142235    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\") pod \"e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") " pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.192786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.192760    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:32.485770 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:32.485741    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4761bc6_7eb1_44bd_8eb6_cd7a07396377.slice/crio-93d958c1ba2ca2612e722971da4735056b657f0c9a064984dd1a18b478a22574 WatchSource:0}: Error finding container 93d958c1ba2ca2612e722971da4735056b657f0c9a064984dd1a18b478a22574: Status 404 returned error can't find the container with id 93d958c1ba2ca2612e722971da4735056b657f0c9a064984dd1a18b478a22574
Nov 01 22:07:32.486233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.486209    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:07:32.488818 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.488786    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:32.488942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.488859    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:32.488942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.488904    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:32.884369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.884280    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:07:32.889226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.889205    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:32.895037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.894999    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-830149-dockercfg-6zx4v\""
Nov 01 22:07:32.903148 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.903120    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:07:32.903435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.903400    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:32.903841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.903820    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947jg\" (UniqueName: \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:32.919857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.919833    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-830149\""
Nov 01 22:07:32.949118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.949087    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:07:32.953819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.953799    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:32.967252 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:32.967217    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:07:33.055319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.041829    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947jg\" (UniqueName: \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:33.055319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.041944    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.055319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.041972    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtgn\" (UniqueName: \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.055319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.042082    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:33.086322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.066374    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:33.178370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.166163    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947jg\" (UniqueName: \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\") pod \"payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:33.178370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.168000    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.178370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.168030    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtgn\" (UniqueName: \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.178370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.168861    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.195371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.193968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtgn\" (UniqueName: \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\") pod \"payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.208873 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.208826    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:33.280131 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.279889    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:33.388478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.388444    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" event={"ID":"b4761bc6-7eb1-44bd-8eb6-cd7a07396377","Type":"ContainerStarted","Data":"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"}
Nov 01 22:07:33.388621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.388510    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" event={"ID":"b4761bc6-7eb1-44bd-8eb6-cd7a07396377","Type":"ContainerStarted","Data":"93d958c1ba2ca2612e722971da4735056b657f0c9a064984dd1a18b478a22574"}
Nov 01 22:07:33.417745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.417684    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podStartSLOduration=2.417666748 podStartE2EDuration="2.417666748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:33.4145932 +0000 UTC m=+1356.670772964" watchObservedRunningTime="2026-11-01 22:07:33.417666748 +0000 UTC m=+1356.673846512"
Nov 01 22:07:33.517121 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:33.517078    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7ce88d_7eac_476e_9a1c_1fad0d0b8e80.slice/crio-ba2e689bdc7e62bd7cf58782a43c385efbc6588122421e28d1637e7a2a362e98 WatchSource:0}: Error finding container ba2e689bdc7e62bd7cf58782a43c385efbc6588122421e28d1637e7a2a362e98: Status 404 returned error can't find the container with id ba2e689bdc7e62bd7cf58782a43c385efbc6588122421e28d1637e7a2a362e98
Nov 01 22:07:33.519699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.519659    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:07:33.602072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:33.602040    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:07:33.602501 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:33.602475    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3a7de6_ee97_44e4_b5ee_2dbc1ac2eba2.slice/crio-20f52139f1415c44471c19fd0890d5ad4a7dcf5d2a2e9b1ae2ae5bc074d0ad63 WatchSource:0}: Error finding container 20f52139f1415c44471c19fd0890d5ad4a7dcf5d2a2e9b1ae2ae5bc074d0ad63: Status 404 returned error can't find the container with id 20f52139f1415c44471c19fd0890d5ad4a7dcf5d2a2e9b1ae2ae5bc074d0ad63
Nov 01 22:07:34.193623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.193588    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:34.195410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.195379    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:34.195542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.195443    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:34.397371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.397333    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" event={"ID":"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80","Type":"ContainerStarted","Data":"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"}
Nov 01 22:07:34.397583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.397400    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" event={"ID":"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80","Type":"ContainerStarted","Data":"ba2e689bdc7e62bd7cf58782a43c385efbc6588122421e28d1637e7a2a362e98"}
Nov 01 22:07:34.397583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.397429    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:34.399516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.399470    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" event={"ID":"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2","Type":"ContainerStarted","Data":"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"}
Nov 01 22:07:34.399654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.399542    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" event={"ID":"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2","Type":"ContainerStarted","Data":"20f52139f1415c44471c19fd0890d5ad4a7dcf5d2a2e9b1ae2ae5bc074d0ad63"}
Nov 01 22:07:34.419433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.419386    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" podStartSLOduration=2.419370963 podStartE2EDuration="2.419370963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:34.415432973 +0000 UTC m=+1357.671612736" watchObservedRunningTime="2026-11-01 22:07:34.419370963 +0000 UTC m=+1357.675550725"
Nov 01 22:07:34.435369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:34.435278    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" podStartSLOduration=2.435266609 podStartE2EDuration="2.435266609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:34.43245168 +0000 UTC m=+1357.688631447" watchObservedRunningTime="2026-11-01 22:07:34.435266609 +0000 UTC m=+1357.691446371"
Nov 01 22:07:34.819811 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:34.819774    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:07:35.193696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:35.193609    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:35.193696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:35.193676    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:35.408233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:35.408198    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:36.193815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:36.193786    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:36.194257 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:36.194232    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:37.178856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.178819    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:07:37.184133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.184103    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.188590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.188561    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0f9b7e-serving-cert\""
Nov 01 22:07:37.193562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.193539    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:07:37.193763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.193703    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:37.194239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.194208    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:37.341741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.341704    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.341924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.341796    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9pz\" (UniqueName: \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.341924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.341847    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.342011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.341998    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.444179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.444085    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.444179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.444171    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pz\" (UniqueName: \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.444552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.444202    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.444552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.444479    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.449320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.445521    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.475381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.468773    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.475381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.470581    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.476415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.476387    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9pz\" (UniqueName: \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\") pod \"maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.508938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.508903    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:37.839657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:37.837390    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:07:38.193694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.193611    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:38.193694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.193672    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:38.497318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.496889    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" event={"ID":"f2b68648-ac51-409d-9a2d-15b17643c226","Type":"ContainerStarted","Data":"39bd7d9d8d3b2ddf028cf0d8884c74082868472a33b64039f08d0385e4892aa2"}
Nov 01 22:07:38.497318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.497012    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" event={"ID":"f2b68648-ac51-409d-9a2d-15b17643c226","Type":"ContainerStarted","Data":"437b184771c1f24d9700184953e551f5d2303a642fbc007577b8ff8d57791b88"}
Nov 01 22:07:38.500273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.498758    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:38.522216 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:38.522163    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" podStartSLOduration=1.063238104 podStartE2EDuration="1.522148519s" totalImagesPullingTime="458.910415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:38.519546355 +0000 UTC m=+1361.775726120" watchObservedRunningTime="2026-11-01 22:07:38.522148519 +0000 UTC m=+1361.778328281"
Nov 01 22:07:39.193062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:39.193025    5845 patch_prober.go:29] interesting pod/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:07:39.193279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:39.193086    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 01 22:07:40.198628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:40.198600    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:40.199089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:40.198914    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:40.199560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:40.199538    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:07:40.868675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:40.868638    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:07:41.315459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.315417    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:07:41.360668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.360633    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:07:41.361343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.361279    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.369584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.369554    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17957f-openshift-default-dockercfg-5jkrs\""
Nov 01 22:07:41.376789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.376762    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.376939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.376927    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377028    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377129    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377207 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377197    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjvz\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377267    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377387    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377572 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377556    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.377755 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.377735    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.479698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.479651    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.479906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.479761    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.479906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.479817    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.479935    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.479975    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjvz\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480036    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480216 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480107    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480244    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480274    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480340    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.480579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.480553    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.481383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.481362    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.481460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.481446    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.483029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.483006    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.483337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.483316    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.492990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.492968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjvz\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.493727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.493708    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:41.686707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:41.686627    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:42.009075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.009044    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:07:42.009941 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:42.009906    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf303a365_4e50_4f59_84fe_f6e83ca4c765.slice/crio-61c96e460bd2df347af7bc2b5c38e366e64efc7745bdf7cd82a8266c63bc5135 WatchSource:0}: Error finding container 61c96e460bd2df347af7bc2b5c38e366e64efc7745bdf7cd82a8266c63bc5135: Status 404 returned error can't find the container with id 61c96e460bd2df347af7bc2b5c38e366e64efc7745bdf7cd82a8266c63bc5135
Nov 01 22:07:42.013525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.013493    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:42.013605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.013563    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:42.013605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.013595    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:07:42.538552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.538514    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" event={"ID":"f303a365-4e50-4f59-84fe-f6e83ca4c765","Type":"ContainerStarted","Data":"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"}
Nov 01 22:07:42.538950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.538574    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" event={"ID":"f303a365-4e50-4f59-84fe-f6e83ca4c765","Type":"ContainerStarted","Data":"61c96e460bd2df347af7bc2b5c38e366e64efc7745bdf7cd82a8266c63bc5135"}
Nov 01 22:07:42.567241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.567185    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podStartSLOduration=1.567172113 podStartE2EDuration="1.567172113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:07:42.56204533 +0000 UTC m=+1365.818225117" watchObservedRunningTime="2026-11-01 22:07:42.567172113 +0000 UTC m=+1365.823351876"
Nov 01 22:07:42.686973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:42.686934    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:43.689398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:43.689363    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:07:43.689815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:43.689429    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 22:07:44.510094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:44.510066    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:07:44.687165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:44.687127    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:07:44.687385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:44.687184    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 22:07:45.411214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:45.411183    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:07:45.687954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:45.687869    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:07:45.687954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:45.687925    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 22:07:45.876482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:45.876427    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerName="payload-pre-processing" containerID="cri-o://a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee" gracePeriod=25
Nov 01 22:07:46.008732 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:46.008705    5845 logging.go:55] [core] [Channel #1151 SubChannel #1152]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.107:50052", ServerName: "10.134.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.107:50052: connect: connection refused"
Nov 01 22:07:46.226608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.226583    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:07:46.397348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.397215    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\" (UniqueName: \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\") pod \"b11f2650-ec48-4e3f-97b9-29f63c78695f\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") "
Nov 01 22:07:46.397640 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.397617    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\") pod \"b11f2650-ec48-4e3f-97b9-29f63c78695f\" (UID: \"b11f2650-ec48-4e3f-97b9-29f63c78695f\") "
Nov 01 22:07:46.398071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.398047    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume" pod "b11f2650-ec48-4e3f-97b9-29f63c78695f" (UID: "b11f2650-ec48-4e3f-97b9-29f63c78695f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:07:46.398370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.398354    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11f2650-ec48-4e3f-97b9-29f63c78695f-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:46.399682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.399661    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw" pod "b11f2650-ec48-4e3f-97b9-29f63c78695f" (UID: "b11f2650-ec48-4e3f-97b9-29f63c78695f"). InnerVolumeSpecName "kube-api-access-q5fbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:07:46.417567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.417548    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:07:46.499923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.499886    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5fbw\" (UniqueName: \"kubernetes.io/projected/b11f2650-ec48-4e3f-97b9-29f63c78695f-kube-api-access-q5fbw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:07:46.613377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.613327    5845 generic.go:356] "Generic (PLEG): container finished" podID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerID="a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee" exitCode=0
Nov 01 22:07:46.613377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.613364    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"
Nov 01 22:07:46.613592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.613404    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" event={"ID":"b11f2650-ec48-4e3f-97b9-29f63c78695f","Type":"ContainerDied","Data":"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"}
Nov 01 22:07:46.613592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.613460    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" event={"ID":"b11f2650-ec48-4e3f-97b9-29f63c78695f","Type":"ContainerDied","Data":"146bd0c692f45ec692b601fc4b2bba4fe60752b95c28b6e11bd8f109e03cfda2"}
Nov 01 22:07:46.613592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.613489    5845 scope.go:122] "RemoveContainer" containerID="a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"
Nov 01 22:07:46.626452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.626439    5845 scope.go:122] "RemoveContainer" containerID="a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"
Nov 01 22:07:46.626727 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:07:46.626706    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee\": container with ID starting with a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee not found: ID does not exist" containerID="a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"
Nov 01 22:07:46.626811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.626737    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee"} err="failed to get container status \"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee\": rpc error: code = NotFound desc = could not find container \"a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee\": container with ID starting with a42318b4eabd4c640fbe1b496a91e59caf1696c640403d5feaac4d9311a05aee not found: ID does not exist"
Nov 01 22:07:46.639476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.639453    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:07:46.643348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.643325    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd"]
Nov 01 22:07:46.687733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.687425    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:07:46.687733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.687481    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 22:07:46.882038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:46.882007    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" path="/var/lib/kubelet/pods/b11f2650-ec48-4e3f-97b9-29f63c78695f/volumes"
Nov 01 22:07:47.009355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:47.009314    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18657f-7f9f69fb5f-gwxwd" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.107:50052\" within 1s: context deadline exceeded"
Nov 01 22:07:47.009528 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:07:47.009365    5845 logging.go:55] [core] [Channel #1151 SubChannel #1152]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.107:50052", ServerName: "10.134.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.107:50052: operation was canceled"
Nov 01 22:07:47.688387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:47.687424    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:07:47.688387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:47.687487    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 01 22:07:48.692229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:48.692198    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:48.692718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:48.692698    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:07:48.693246 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:07:48.693227    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:08:04.828836 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:04.828796    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:08:11.276764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.276732    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:08:11.277987 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.277952    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerName="payload-pre-processing"
Nov 01 22:08:11.278231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.277994    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerName="payload-pre-processing"
Nov 01 22:08:11.278326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.278271    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11f2650-ec48-4e3f-97b9-29f63c78695f" containerName="payload-pre-processing"
Nov 01 22:08:11.284780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.284755    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.302764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.302558    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:08:11.364869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.364837    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365021 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.364965    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365059    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365137    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365222 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365202    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365236    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365352    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365391    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.365489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.365477    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4l6\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467257    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467322    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467368    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4l6\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467468    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467534    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467605    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467664    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467704    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467738    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.467887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.467823    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.468044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.468020    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.468108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.468083    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.468186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.468171    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.468369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.468346    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.469692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.469673    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.470033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.470010    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.508229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.508198    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4l6\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.508393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.508244    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\") pod \"e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") " pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.607428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.607341    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:11.910788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.910760    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:08:11.911480 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:11.911451    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a04513_864a_4c2f_be26_be04673400a7.slice/crio-6304d10e388b6d403265eb7d3288e5227140f82367a01a57ba3b3f3b57e2bc09 WatchSource:0}: Error finding container 6304d10e388b6d403265eb7d3288e5227140f82367a01a57ba3b3f3b57e2bc09: Status 404 returned error can't find the container with id 6304d10e388b6d403265eb7d3288e5227140f82367a01a57ba3b3f3b57e2bc09
Nov 01 22:08:11.914794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.914764    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:11.914876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.914829    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:11.914876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.914858    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:11.934317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.934278    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:08:11.940597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.940579    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:11.948693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.948671    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8483c2-dockercfg-s4gb8\""
Nov 01 22:08:11.948821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.948747    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8483c2\""
Nov 01 22:08:11.949696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.949675    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:08:11.980129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.975900    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pfg\" (UniqueName: \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:11.980129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.976150    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:11.987474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.987449    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:08:11.992269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:11.992250    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.002290 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.002265    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:08:12.092523 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.080099    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.111713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.111659    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.112108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.112084    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.112341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.112291    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pfg\" (UniqueName: \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.124166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.094660    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.147410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.147311    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pfg\" (UniqueName: \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\") pod \"payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.237997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.237949    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.238409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.238367    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.265761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.265725    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.284740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.283611    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.284740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.283754    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\") pod \"payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.320253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.320224    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.586020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.585087    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:08:12.679206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.679171    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:08:12.891003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.890925    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" event={"ID":"37272cfa-7de1-4bdf-bd98-2ded556f162a","Type":"ContainerStarted","Data":"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"}
Nov 01 22:08:12.891003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.890987    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" event={"ID":"37272cfa-7de1-4bdf-bd98-2ded556f162a","Type":"ContainerStarted","Data":"2e1ea825951a68589f24e86747b0c8b7a598258f102f8e2d4e065e88dee16619"}
Nov 01 22:08:12.891224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.891067    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:12.893092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.893063    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" event={"ID":"22a04513-864a-4c2f-be26-be04673400a7","Type":"ContainerStarted","Data":"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"}
Nov 01 22:08:12.893227 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.893116    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" event={"ID":"22a04513-864a-4c2f-be26-be04673400a7","Type":"ContainerStarted","Data":"6304d10e388b6d403265eb7d3288e5227140f82367a01a57ba3b3f3b57e2bc09"}
Nov 01 22:08:12.894744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.894720    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" event={"ID":"339a5d91-271c-4509-b485-908329cdc38c","Type":"ContainerStarted","Data":"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"}
Nov 01 22:08:12.894850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.894757    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" event={"ID":"339a5d91-271c-4509-b485-908329cdc38c","Type":"ContainerStarted","Data":"e20f361da7aa39250bd2ea1893813af0a09480094cbce80f00f28f46ab8b24bb"}
Nov 01 22:08:12.894850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.894794    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:12.912489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.912446    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" podStartSLOduration=1.912433526 podStartE2EDuration="1.912433526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:12.910045336 +0000 UTC m=+1396.166225102" watchObservedRunningTime="2026-11-01 22:08:12.912433526 +0000 UTC m=+1396.168613290"
Nov 01 22:08:12.936462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.936424    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" podStartSLOduration=1.936413456 podStartE2EDuration="1.936413456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:12.931615244 +0000 UTC m=+1396.187795008" watchObservedRunningTime="2026-11-01 22:08:12.936413456 +0000 UTC m=+1396.192593219"
Nov 01 22:08:12.962870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:12.962831    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podStartSLOduration=1.96281924 podStartE2EDuration="1.96281924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:12.95858529 +0000 UTC m=+1396.214765054" watchObservedRunningTime="2026-11-01 22:08:12.96281924 +0000 UTC m=+1396.218999003"
Nov 01 22:08:13.608680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:13.608645    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:13.633561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:13.633510    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:13.633899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:13.633864    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:14.608137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:14.608103    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:14.608349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:14.608157    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:15.608371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:15.608337    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:15.608880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:15.608401    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:15.935497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:15.935416    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:08:15.935673 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:15.935622    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" podUID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" containerName="maas-api" containerID="cri-o://e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2" gracePeriod=30
Nov 01 22:08:16.610037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.609609    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:16.610037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.609666    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:16.685340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.685267    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:08:16.816836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.816805    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\" (UniqueName: \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\") pod \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") "
Nov 01 22:08:16.817112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.817095    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\") pod \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") "
Nov 01 22:08:16.817213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.817178    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\") pod \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") "
Nov 01 22:08:16.817283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.817214    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\") pod \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\" (UID: \"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143\") "
Nov 01 22:08:16.817830 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.817802    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca" pod "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" (UID: "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:08:16.818123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.818104    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:08:16.819539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.819508    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn" pod "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" (UID: "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143"). InnerVolumeSpecName "kube-api-access-5wjkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:08:16.819676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.819648    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls" pod "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" (UID: "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:08:16.820052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.820026    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs" pod "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" (UID: "84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:08:16.922747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.921046    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:08:16.922747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.921086    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:08:16.922747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:16.921103    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjkn\" (UniqueName: \"kubernetes.io/projected/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143-kube-api-access-5wjkn\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:08:17.026374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.026342    5845 generic.go:356] "Generic (PLEG): container finished" podID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" containerID="e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2" exitCode=0
Nov 01 22:08:17.026539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.026409    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"
Nov 01 22:08:17.026539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.026420    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" event={"ID":"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143","Type":"ContainerDied","Data":"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"}
Nov 01 22:08:17.026539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.026496    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx" event={"ID":"84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143","Type":"ContainerDied","Data":"3a9e3890bb94b64c79c7aa1eb1958ccb13d4cf9bb60146836cea147f2230e281"}
Nov 01 22:08:17.026539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.026520    5845 scope.go:122] "RemoveContainer" containerID="e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"
Nov 01 22:08:17.038375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.038358    5845 scope.go:122] "RemoveContainer" containerID="e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"
Nov 01 22:08:17.038678 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:17.038650    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2\": container with ID starting with e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2 not found: ID does not exist" containerID="e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"
Nov 01 22:08:17.038719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.038690    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2"} err="failed to get container status \"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2\": rpc error: code = NotFound desc = could not find container \"e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2\": container with ID starting with e9869697d05616bff4a98d3ac2e2d5ca44edaee4a5956c7a1273cabc66a215e2 not found: ID does not exist"
Nov 01 22:08:17.089179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.089148    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:08:17.099241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.099207    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18657f-6c5dd7f487-nxddx"]
Nov 01 22:08:17.608547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.608511    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:17.608720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:17.608572    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:18.608255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:18.608224    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:08:18.608705 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:18.608273    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 01 22:08:18.930371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:18.930273    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" path="/var/lib/kubelet/pods/84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143/volumes"
Nov 01 22:08:19.613812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:19.613777    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:19.614277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:19.614093    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:19.614926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:19.614908    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:08:23.704006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.703963    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:08:23.705372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.705340    5845 cpu_manager.go:414] "containerMap: removing container" podUID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" containerName="maas-api"
Nov 01 22:08:23.705372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.705365    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" containerName="maas-api"
Nov 01 22:08:23.705776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.705656    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b66b0b-e7e1-4d42-b1fd-c39dc6f7b143" containerName="maas-api"
Nov 01 22:08:23.711142 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.711124    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.723584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.723541    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e2a91d-serving-cert\""
Nov 01 22:08:23.728020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.727993    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:08:23.800279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.800220    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.800473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.800371    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.800675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.800653    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc9js\" (UniqueName: \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.800893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.800877    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.905822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.905782    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc9js\" (UniqueName: \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.906575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.906548    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.907405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.907373    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.908498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.908477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.909376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.908327    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.977396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.972075    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.977396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.972546    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:23.977396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:23.974283    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc9js\" (UniqueName: \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\") pod \"maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:24.021783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:24.021740    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:08:24.024485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:24.023678    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:08:24.060089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:24.060054    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:24.516660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:24.515256    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:08:24.520673 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:24.520640    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6d1743_0225_4e1e_a4d3_386b89777915.slice/crio-e6502b2931a6e587fb8c40f58fbfdedf01e04894819b68b0addfe9dab576744d WatchSource:0}: Error finding container e6502b2931a6e587fb8c40f58fbfdedf01e04894819b68b0addfe9dab576744d: Status 404 returned error can't find the container with id e6502b2931a6e587fb8c40f58fbfdedf01e04894819b68b0addfe9dab576744d
Nov 01 22:08:25.200782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:25.200741    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" event={"ID":"cb6d1743-0225-4e1e-a4d3-386b89777915","Type":"ContainerStarted","Data":"e4e79d4f227ba697b108fc3fe125f19d26c2d07d2261f29f7f1e841770327bd0"}
Nov 01 22:08:25.201292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:25.200809    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" event={"ID":"cb6d1743-0225-4e1e-a4d3-386b89777915","Type":"ContainerStarted","Data":"e6502b2931a6e587fb8c40f58fbfdedf01e04894819b68b0addfe9dab576744d"}
Nov 01 22:08:25.201931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:25.201905    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:25.264138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:25.264076    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" podStartSLOduration=1.743235833 podStartE2EDuration="2.264062658s" totalImagesPullingTime="520.826825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:25.261862048 +0000 UTC m=+1408.518041811" watchObservedRunningTime="2026-11-01 22:08:25.264062658 +0000 UTC m=+1408.520242421"
Nov 01 22:08:31.249793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.249750    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:08:31.255041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.255016    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.266264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.266227    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:08:31.294689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.294655    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-487639\""
Nov 01 22:08:31.313585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.313557    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-487639-dockercfg-ftx4j\""
Nov 01 22:08:31.331203 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.331172    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4vj7\" (UniqueName: \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.331649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.331619    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.442319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.439648    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.442319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.439829    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4vj7\" (UniqueName: \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.442319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.441356    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.471374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.468968    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4vj7\" (UniqueName: \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\") pod \"payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.596373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.594852    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:31.945499 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:31.945465    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:08:31.946366 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:31.946334    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c119c05_2a1c_4e9a_ba8f_f6b9769458c2.slice/crio-1fa13a71ff068f7d88e00d02447335feae31f9afb50b22d995be6607e905b6e9 WatchSource:0}: Error finding container 1fa13a71ff068f7d88e00d02447335feae31f9afb50b22d995be6607e905b6e9: Status 404 returned error can't find the container with id 1fa13a71ff068f7d88e00d02447335feae31f9afb50b22d995be6607e905b6e9
Nov 01 22:08:32.252311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:32.252274    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:08:32.302733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:32.302697    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" event={"ID":"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2","Type":"ContainerStarted","Data":"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"}
Nov 01 22:08:32.302890 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:32.302755    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" event={"ID":"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2","Type":"ContainerStarted","Data":"1fa13a71ff068f7d88e00d02447335feae31f9afb50b22d995be6607e905b6e9"}
Nov 01 22:08:32.303662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:32.303636    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:32.323388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:32.323335    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" podStartSLOduration=1.3233183259999999 podStartE2EDuration="1.323318326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:32.321136447 +0000 UTC m=+1415.577316209" watchObservedRunningTime="2026-11-01 22:08:32.323318326 +0000 UTC m=+1415.579498092"
Nov 01 22:08:34.819055 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:34.819016    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:08:35.059408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.059378    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:08:35.065260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.065228    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.073359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.073283    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8483c2-serving-cert\""
Nov 01 22:08:35.076676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.076630    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:08:35.140335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.140291    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.140515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.140466    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.140726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.140698    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmh8\" (UniqueName: \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.140809 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.140793    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.241995 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.241958    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.242205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.242159    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmh8\" (UniqueName: \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.242281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.242207    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.242364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.242282    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.244352 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:35.242635    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-serving-cert: secret "maas-api-e2e-inf-b-8483c2-serving-cert" not found
Nov 01 22:08:35.244352 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:35.242763    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls podName:58029f31-95cf-4ff1-8ba9-ac769afa0833 nodeName:}" failed. No retries permitted until 2026-11-01 22:08:35.742735453 +0000 UTC m=+1418.998915196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls") pod "maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833") : secret "maas-api-e2e-inf-b-8483c2-serving-cert" not found
Nov 01 22:08:35.244352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.243134    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.246157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.246134    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.259663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.259634    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmh8\" (UniqueName: \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.767607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:35.762226    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:35.767607 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:35.762828    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-serving-cert: secret "maas-api-e2e-inf-b-8483c2-serving-cert" not found
Nov 01 22:08:35.767607 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:08:35.762901    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls podName:58029f31-95cf-4ff1-8ba9-ac769afa0833 nodeName:}" failed. No retries permitted until 2026-11-01 22:08:36.762881906 +0000 UTC m=+1420.019061664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls") pod "maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833") : secret "maas-api-e2e-inf-b-8483c2-serving-cert" not found
Nov 01 22:08:36.780767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:36.780730    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:36.783353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:36.783327    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"maas-api-e2e-inf-b-8483c2-548854584c-rjfhr\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:36.893785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:36.893752    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:37.198588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:37.198502    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:08:37.203168 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:37.203138    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58029f31_95cf_4ff1_8ba9_ac769afa0833.slice/crio-9af97aaf8ef380cecab0134c52298c8b6efef2073edac3ea264cb2cce176f93f WatchSource:0}: Error finding container 9af97aaf8ef380cecab0134c52298c8b6efef2073edac3ea264cb2cce176f93f: Status 404 returned error can't find the container with id 9af97aaf8ef380cecab0134c52298c8b6efef2073edac3ea264cb2cce176f93f
Nov 01 22:08:37.355356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:37.355319    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" event={"ID":"58029f31-95cf-4ff1-8ba9-ac769afa0833","Type":"ContainerStarted","Data":"9af97aaf8ef380cecab0134c52298c8b6efef2073edac3ea264cb2cce176f93f"}
Nov 01 22:08:38.365342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:38.365289    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" event={"ID":"58029f31-95cf-4ff1-8ba9-ac769afa0833","Type":"ContainerStarted","Data":"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"}
Nov 01 22:08:38.365812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:38.365385    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:38.389880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:38.389834    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" podStartSLOduration=2.97830205 podStartE2EDuration="3.389821092s" totalImagesPullingTime="411.519042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:38.385774984 +0000 UTC m=+1421.641954760" watchObservedRunningTime="2026-11-01 22:08:38.389821092 +0000 UTC m=+1421.646000856"
Nov 01 22:08:42.232622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.232589    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:08:42.238455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.238430    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.269672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.269638    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:08:42.329449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.329419    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9d8b5d-openshift-default-dockercfg-pxzwg\""
Nov 01 22:08:42.377990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.377958    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378007    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378126    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378229    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378283    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378371    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378444    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378465    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nlg\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.378496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.378488    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.479980 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.479950    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.479999    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480053    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480145    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480185    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480237    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480365    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480392    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nlg\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480429    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480615    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480670    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.480911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.480887    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.481122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.481096    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.481385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.481364    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.482465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.482445    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.483081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.483032    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.493090 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.493062    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nlg\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.493182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.493090    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.567706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.567663    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:42.902671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.902642    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:08:42.903046 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:42.903020    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99efaa10_3ca4_4a9d_98f0_d776ffb1a467.slice/crio-feee922654db1305418006ca0b3f0cd496767ca4dda7b8d5b8fbdd803d9530af WatchSource:0}: Error finding container feee922654db1305418006ca0b3f0cd496767ca4dda7b8d5b8fbdd803d9530af: Status 404 returned error can't find the container with id feee922654db1305418006ca0b3f0cd496767ca4dda7b8d5b8fbdd803d9530af
Nov 01 22:08:42.905975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.905941    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:42.906058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.906007    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:42.906058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:42.906035    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:08:43.433775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.432526    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" event={"ID":"99efaa10-3ca4-4a9d-98f0-d776ffb1a467","Type":"ContainerStarted","Data":"579a2e7ee54d2ddad0e3f2d5027e4670a06799ee3bf3116c8a39eb52e38af40c"}
Nov 01 22:08:43.433775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.432635    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" event={"ID":"99efaa10-3ca4-4a9d-98f0-d776ffb1a467","Type":"ContainerStarted","Data":"feee922654db1305418006ca0b3f0cd496767ca4dda7b8d5b8fbdd803d9530af"}
Nov 01 22:08:43.465058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.464984    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podStartSLOduration=1.464963481 podStartE2EDuration="1.464963481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:43.455857578 +0000 UTC m=+1426.712037359" watchObservedRunningTime="2026-11-01 22:08:43.464963481 +0000 UTC m=+1426.721143245"
Nov 01 22:08:43.568860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.568827    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:43.571113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.571082    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:43.571235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:43.571142    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:44.331675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:44.331641    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:08:44.396501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:44.396471    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:08:44.574108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:44.573708    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:44.574108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:44.573783    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:45.569719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:45.569679    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:45.569928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:45.569744    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:46.569794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:46.569755    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:46.570237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:46.569818    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:47.576321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:47.571687    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:47.576321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:47.571764    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:48.570127 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:48.570092    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 01 22:08:48.570347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:48.570169    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 01 22:08:49.593940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:49.593902    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:49.594441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:49.594414    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:49.596006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:49.595981    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:08:56.424245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.424208    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:08:56.430870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.430846    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.442933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.442896    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:08:56.444531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.444504    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dfdb11\""
Nov 01 22:08:56.444712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.444587    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dfdb11-dockercfg-2fjpv\""
Nov 01 22:08:56.491981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.491948    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:08:56.497121 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.497090    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.523600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.523571    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:08:56.558053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.558011    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kspf8\" (UniqueName: \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.558269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.558220    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvwj\" (UniqueName: \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.558373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.558267    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.558431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.558413    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.663836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.663797    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvwj\" (UniqueName: \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.664128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.664099    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.664440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.664421    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.664853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.664828    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kspf8\" (UniqueName: \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.672024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.672001    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.716185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.716104    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.733777 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.726945    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kspf8\" (UniqueName: \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\") pod \"payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.748149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.748116    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvwj\" (UniqueName: \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\") pod \"payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:56.768627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.768592    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:56.828352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:56.825137    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:57.126793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.126765    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:08:57.127244 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:57.127215    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287faada_dc3c_4cd5_97e4_492288419e8e.slice/crio-34919abcc9f8eba32d9e892410f07ced0cd4a935085449a611568d57580742e2 WatchSource:0}: Error finding container 34919abcc9f8eba32d9e892410f07ced0cd4a935085449a611568d57580742e2: Status 404 returned error can't find the container with id 34919abcc9f8eba32d9e892410f07ced0cd4a935085449a611568d57580742e2
Nov 01 22:08:57.195553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.195524    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:08:57.196022 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:08:57.196002    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-2bdea82a14ae1bb0fb3a67bd173e16d02c6a8b51c61100f03ccad9a8594f3acf WatchSource:0}: Error finding container 2bdea82a14ae1bb0fb3a67bd173e16d02c6a8b51c61100f03ccad9a8594f3acf: Status 404 returned error can't find the container with id 2bdea82a14ae1bb0fb3a67bd173e16d02c6a8b51c61100f03ccad9a8594f3acf
Nov 01 22:08:57.734526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.734487    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" event={"ID":"40cdf802-c25f-446d-ae2b-26a71629e212","Type":"ContainerStarted","Data":"334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663"}
Nov 01 22:08:57.734977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.734553    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" event={"ID":"40cdf802-c25f-446d-ae2b-26a71629e212","Type":"ContainerStarted","Data":"2bdea82a14ae1bb0fb3a67bd173e16d02c6a8b51c61100f03ccad9a8594f3acf"}
Nov 01 22:08:57.734977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.734582    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:08:57.736697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.736669    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" event={"ID":"287faada-dc3c-4cd5-97e4-492288419e8e","Type":"ContainerStarted","Data":"052764ff07a31452fb08bf7714fe8052bf1010a8c7354fe05609003e70c7f0a9"}
Nov 01 22:08:57.736826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.736721    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" event={"ID":"287faada-dc3c-4cd5-97e4-492288419e8e","Type":"ContainerStarted","Data":"34919abcc9f8eba32d9e892410f07ced0cd4a935085449a611568d57580742e2"}
Nov 01 22:08:57.736826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.736749    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:08:57.758927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.758873    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" podStartSLOduration=1.758856544 podStartE2EDuration="1.758856544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:57.752211405 +0000 UTC m=+1441.008391170" watchObservedRunningTime="2026-11-01 22:08:57.758856544 +0000 UTC m=+1441.015036313"
Nov 01 22:08:57.774998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:08:57.774957    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" podStartSLOduration=1.77494631 podStartE2EDuration="1.77494631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:08:57.770857301 +0000 UTC m=+1441.027037064" watchObservedRunningTime="2026-11-01 22:08:57.77494631 +0000 UTC m=+1441.031126073"
Nov 01 22:09:04.818557 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:09:04.818514    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:09:08.754341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:08.754246    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:09:08.755068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:08.755044    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:09:10.535071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:10.535034    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:09:15.535813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.535761    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" podUID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" containerName="payload-processing" containerID="cri-o://7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a" gracePeriod=25
Nov 01 22:09:15.854620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.854589    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:09:15.947480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.947448    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\" (UniqueName: \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\") pod \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") "
Nov 01 22:09:15.947636 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.947572    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\") pod \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\" (UID: \"1d552b5b-7b02-467c-9f79-6d53a1125cb5\") "
Nov 01 22:09:15.947961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.947937    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume" pod "1d552b5b-7b02-467c-9f79-6d53a1125cb5" (UID: "1d552b5b-7b02-467c-9f79-6d53a1125cb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:09:15.948503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.948482    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d552b5b-7b02-467c-9f79-6d53a1125cb5-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:09:15.949933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.949913    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n" pod "1d552b5b-7b02-467c-9f79-6d53a1125cb5" (UID: "1d552b5b-7b02-467c-9f79-6d53a1125cb5"). InnerVolumeSpecName "kube-api-access-9bc9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:09:15.953623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.953600    5845 generic.go:356] "Generic (PLEG): container finished" podID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" containerID="7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a" exitCode=0
Nov 01 22:09:15.953711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.953631    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" event={"ID":"1d552b5b-7b02-467c-9f79-6d53a1125cb5","Type":"ContainerDied","Data":"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"}
Nov 01 22:09:15.953711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.953666    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg" event={"ID":"1d552b5b-7b02-467c-9f79-6d53a1125cb5","Type":"ContainerDied","Data":"492fec0c7cd607bfae07a51bfb571d393214dc5933927d9f4398d9b83948497c"}
Nov 01 22:09:15.953711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.953671    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"
Nov 01 22:09:15.953711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.953687    5845 scope.go:122] "RemoveContainer" containerID="7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"
Nov 01 22:09:15.974949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.974933    5845 scope.go:122] "RemoveContainer" containerID="7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"
Nov 01 22:09:15.975225 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:09:15.975206    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a\": container with ID starting with 7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a not found: ID does not exist" containerID="7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"
Nov 01 22:09:15.975282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.975234    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a"} err="failed to get container status \"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a\": rpc error: code = NotFound desc = could not find container \"7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a\": container with ID starting with 7750dc1f5a36cee40315117c904a9620b244bdb10fd98c27820639f18201e22a not found: ID does not exist"
Nov 01 22:09:15.993759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:15.993730    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:09:16.000613 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.000589    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0f9b7e-7456c8d5cb-dfzvg"]
Nov 01 22:09:16.050161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.050084    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bc9n\" (UniqueName: \"kubernetes.io/projected/1d552b5b-7b02-467c-9f79-6d53a1125cb5-kube-api-access-9bc9n\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:09:16.605666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.603845    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:09:16.636319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.633764    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" containerName="payload-processing"
Nov 01 22:09:16.636319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.633810    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" containerName="payload-processing"
Nov 01 22:09:16.636319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.634047    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" containerName="payload-processing"
Nov 01 22:09:16.660609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.660567    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.666591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.664070    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:09:16.691677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.677656    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3e300c-openshift-default-dockercfg-lpvl5\""
Nov 01 22:09:16.809030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.808984    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809140    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809177    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809202    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809251    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809353    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809511    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w649\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809558    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.809652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.809638    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.868510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.868416    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d552b5b-7b02-467c-9f79-6d53a1125cb5" path="/var/lib/kubelet/pods/1d552b5b-7b02-467c-9f79-6d53a1125cb5/volumes"
Nov 01 22:09:16.911189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911154    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.911518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911271    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.911518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911411    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w649\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.911729 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911684    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.911861 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911816    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.911971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.911948    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912088    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912119    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912142    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912176    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912172    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912572 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912553    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912761    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.912876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.912855    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.914557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.914531    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.914794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.914776    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.929709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.929658    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w649\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:16.929848 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:16.929809    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:17.007862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.007833    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:17.305002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.304973    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:09:17.305345 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:09:17.305317    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48f8f3a_093e_43f4_84cc_cbd03315ef3f.slice/crio-e78db7d84684a47156965a6c9673dc5ed3e1aa8ee4308b8d8254ef6527433667 WatchSource:0}: Error finding container e78db7d84684a47156965a6c9673dc5ed3e1aa8ee4308b8d8254ef6527433667: Status 404 returned error can't find the container with id e78db7d84684a47156965a6c9673dc5ed3e1aa8ee4308b8d8254ef6527433667
Nov 01 22:09:17.308799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.308763    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:17.308908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.308845    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:17.308908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.308890    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:17.975521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.975477    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerStarted","Data":"a86a5daa0974e76fd6554636de7018179747532175030083000f29c213ecbdf3"}
Nov 01 22:09:17.976025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:17.975552    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerStarted","Data":"e78db7d84684a47156965a6c9673dc5ed3e1aa8ee4308b8d8254ef6527433667"}
Nov 01 22:09:18.005139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:18.005083    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podStartSLOduration=2.005068256 podStartE2EDuration="2.005068256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:18.001517745 +0000 UTC m=+1461.257697507" watchObservedRunningTime="2026-11-01 22:09:18.005068256 +0000 UTC m=+1461.261248019"
Nov 01 22:09:18.008682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:18.008657    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:18.015206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:18.015182    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:18.984009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:18.983978    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:18.984978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:18.984960    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:09:21.731626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:21.731585    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:09:21.738626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:21.738601    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:21.751258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:21.750543    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:09:21.788715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:21.788675    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3e300c\""
Nov 01 22:09:21.788908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:21.788874    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3e300c-dockercfg-xfk26\""
Nov 01 22:09:22.032627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.028030    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.032627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.028129    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbsh\" (UniqueName: \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.129601 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.129564    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbsh\" (UniqueName: \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.129883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.129870    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.131514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.131491    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.142526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.142501    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbsh\" (UniqueName: \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\") pod \"payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.321933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.321843    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:22.607463 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:09:22.607430    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf3023a_050b_4a7b_a93b_047a3328e1e9.slice/crio-957533202b7fd32025f8bf69f5538036bda5206d8e901f7bddb88990875ac0ab WatchSource:0}: Error finding container 957533202b7fd32025f8bf69f5538036bda5206d8e901f7bddb88990875ac0ab: Status 404 returned error can't find the container with id 957533202b7fd32025f8bf69f5538036bda5206d8e901f7bddb88990875ac0ab
Nov 01 22:09:22.607688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:22.607662    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:09:23.082517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:23.082482    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" event={"ID":"ccf3023a-050b-4a7b-a93b-047a3328e1e9","Type":"ContainerStarted","Data":"a70f67a2b087dae310823301707f9041d15fd6dfd76b0d80b1db5934e38c4187"}
Nov 01 22:09:23.082985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:23.082544    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" event={"ID":"ccf3023a-050b-4a7b-a93b-047a3328e1e9","Type":"ContainerStarted","Data":"957533202b7fd32025f8bf69f5538036bda5206d8e901f7bddb88990875ac0ab"}
Nov 01 22:09:23.082985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:23.082575    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:23.124316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:23.106454    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" podStartSLOduration=2.106436706 podStartE2EDuration="2.106436706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:23.101672916 +0000 UTC m=+1466.357852681" watchObservedRunningTime="2026-11-01 22:09:23.106436706 +0000 UTC m=+1466.362616469"
Nov 01 22:09:27.271491 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.271452    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:09:27.277814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.277789    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.290741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.290714    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:09:27.371177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371129    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxlc\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371414    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371573    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371611    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371667    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371770    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371814    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.371958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.371945    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.372060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.372046    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473579    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473640    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473672    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473747    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473808    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.473945    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.474061    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.474100    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxlc\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.474272    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.474881    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.477323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.475132    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.484325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.480801    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.484325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.481136    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.486213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.486188    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.510796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.510069    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.577378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.520532    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxlc\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.577378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.529541    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.586319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.584206    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\") pod \"e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") " pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:27.597683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:27.596409    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:28.422413 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:28.422381    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:09:28.424430 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:09:28.424402    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0c0a23_451c_43e2_a709_d123d0dd63fb.slice/crio-f4be840e8828251d547b6cae9709294d86ab374f412f7c7d38f1932aa45dfed5 WatchSource:0}: Error finding container f4be840e8828251d547b6cae9709294d86ab374f412f7c7d38f1932aa45dfed5: Status 404 returned error can't find the container with id f4be840e8828251d547b6cae9709294d86ab374f412f7c7d38f1932aa45dfed5
Nov 01 22:09:28.427839 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:28.427806    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:28.427949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:28.427888    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:28.427949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:28.427933    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:29.768818 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:29.768772    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" event={"ID":"ab0c0a23-451c-43e2-a709-d123d0dd63fb","Type":"ContainerStarted","Data":"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"}
Nov 01 22:09:29.770097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:29.770065    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" event={"ID":"ab0c0a23-451c-43e2-a709-d123d0dd63fb","Type":"ContainerStarted","Data":"f4be840e8828251d547b6cae9709294d86ab374f412f7c7d38f1932aa45dfed5"}
Nov 01 22:09:29.990359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:29.956491    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podStartSLOduration=2.95646884 podStartE2EDuration="2.95646884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:29.851234031 +0000 UTC m=+1473.107413807" watchObservedRunningTime="2026-11-01 22:09:29.95646884 +0000 UTC m=+1473.212648603"
Nov 01 22:09:29.990359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:29.978391    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:30.001323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:29.993387    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:30.029880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.029783    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:09:30.082413 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.082367    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 22:09:30.154421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.154375    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:30.603320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.584487    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:30.629380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.611784    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:30.629380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.612116    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:30.629380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.612163    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:30.629380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.613849    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:09:30.735654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.717215    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:30.997319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:30.996601    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.003475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.003442    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:09:31.058112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.055985    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:09:31.174751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.156385    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.244385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.232804    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.463087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.462986    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 01 22:09:31.492509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.492464    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.511943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.499408    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:09:31.511943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.499480    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:09:31.642619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.617420    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.645663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.645632    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:09:31.699389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.665527    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:09:31.699389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.669706    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:31.699389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.669748    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:31.701627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.699840    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:09:31.811326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.788776    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:31.837319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:31.813255    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:32.005323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.000462    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:32.091218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.091123    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:09:32.194619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.091471    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:32.205468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.091543    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:32.264337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.249253    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:09:32.264337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.252469    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:09:32.264337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.252912    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.563327    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.583906    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.584694    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.585991    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.586467    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:09:32.595329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.588402    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:09:32.599582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.599149    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:32.599582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:32.599200    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:33.599237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:33.599200    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:33.599806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:33.599262    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:34.092766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:34.092736    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:09:34.599087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:34.599056    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:34.599381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:34.599319    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:34.834686 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:09:34.834657    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:09:35.597944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:35.597893    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:35.598180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:35.597968    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:36.597899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:36.597864    5845 patch_prober.go:29] interesting pod/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 01 22:09:36.598309 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:36.597938    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 01 22:09:37.603199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:37.603164    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:37.603742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:37.603720    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:37.604396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:37.604374    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:09:40.477848 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.477812    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:09:40.483494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.483468    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.498927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.498872    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-487639-openshift-default-dockercfg-hcp7z\""
Nov 01 22:09:40.505930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.505903    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:09:40.651745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.651711    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.651950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.651772    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.651950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.651893    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svlw\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.651950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.651915    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.652143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.651957    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.652143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.652016    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.652143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.652033    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.652143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.652075    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.652377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.652143    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.753967    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754085    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754114    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754182    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754263    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754428    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754448    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754491    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754525    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754620    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svlw\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754651    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.754808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754723    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.755013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.754992    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.755376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.755355    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.757759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.757735    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.757884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.757861    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.770938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.770911    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.771058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.770948    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svlw\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\") pod \"e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") " pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:40.805498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:40.805461    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:41.154821 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:09:41.154789    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45395db_02f5_4997_ba4a_82626497f3d0.slice/crio-2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a WatchSource:0}: Error finding container 2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a: Status 404 returned error can't find the container with id 2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a
Nov 01 22:09:41.155577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.155554    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:09:41.158986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.158953    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:41.159102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.159016    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:41.159102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.159046    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:09:41.611897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.611850    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" event={"ID":"e45395db-02f5-4997-ba4a-82626497f3d0","Type":"ContainerStarted","Data":"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"}
Nov 01 22:09:41.612508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.612484    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" event={"ID":"e45395db-02f5-4997-ba4a-82626497f3d0","Type":"ContainerStarted","Data":"2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a"}
Nov 01 22:09:41.637467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.637407    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podStartSLOduration=1.637393782 podStartE2EDuration="1.637393782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:09:41.635480622 +0000 UTC m=+1484.891660388" watchObservedRunningTime="2026-11-01 22:09:41.637393782 +0000 UTC m=+1484.893573544"
Nov 01 22:09:41.806119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:41.806087    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:42.808310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:42.808264    5845 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 22:09:42.808787 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:42.808347    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 22:09:43.805793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:43.805758    5845 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 22:09:43.806053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:43.805816    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 22:09:44.850332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:44.843564    5845 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 22:09:44.850332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:44.843625    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 22:09:45.806498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:45.806460    5845 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 22:09:45.806711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:45.806526    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 22:09:46.806478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:46.806442    5845 patch_prober.go:29] interesting pod/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body=
Nov 01 22:09:46.806972 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:46.806502    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused"
Nov 01 22:09:47.811245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:47.811215    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:47.811783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:47.811762    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:47.812434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:47.812413    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:09:57.120946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:57.120920    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:09:57.126074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:09:57.126054    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:10:02.700635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:02.700600    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:10:02.701053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:02.700845    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" podUID="f2b68648-ac51-409d-9a2d-15b17643c226" containerName="maas-api" containerID="cri-o://39bd7d9d8d3b2ddf028cf0d8884c74082868472a33b64039f08d0385e4892aa2" gracePeriod=30
Nov 01 22:10:03.001350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.001151    5845 generic.go:356] "Generic (PLEG): container finished" podID="f2b68648-ac51-409d-9a2d-15b17643c226" containerID="39bd7d9d8d3b2ddf028cf0d8884c74082868472a33b64039f08d0385e4892aa2" exitCode=0
Nov 01 22:10:03.001558 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.001426    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" event={"ID":"f2b68648-ac51-409d-9a2d-15b17643c226","Type":"ContainerDied","Data":"39bd7d9d8d3b2ddf028cf0d8884c74082868472a33b64039f08d0385e4892aa2"}
Nov 01 22:10:03.295858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.295827    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:10:03.336463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.336433    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:10:03.337618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.337593    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f2b68648-ac51-409d-9a2d-15b17643c226" containerName="maas-api"
Nov 01 22:10:03.337618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.337616    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b68648-ac51-409d-9a2d-15b17643c226" containerName="maas-api"
Nov 01 22:10:03.337818 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.337809    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b68648-ac51-409d-9a2d-15b17643c226" containerName="maas-api"
Nov 01 22:10:03.341739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.341716    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.346409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.346383    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-830149-openshift-default-dockercfg-kkt65\""
Nov 01 22:10:03.356132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.356104    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:10:03.393485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.393454    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\" (UniqueName: \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\") pod \"f2b68648-ac51-409d-9a2d-15b17643c226\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") "
Nov 01 22:10:03.393654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.393541    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\") pod \"f2b68648-ac51-409d-9a2d-15b17643c226\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") "
Nov 01 22:10:03.393778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.393759    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\") pod \"f2b68648-ac51-409d-9a2d-15b17643c226\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") "
Nov 01 22:10:03.393947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.393928    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\") pod \"f2b68648-ac51-409d-9a2d-15b17643c226\" (UID: \"f2b68648-ac51-409d-9a2d-15b17643c226\") "
Nov 01 22:10:03.394029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.393943    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca" pod "f2b68648-ac51-409d-9a2d-15b17643c226" (UID: "f2b68648-ac51-409d-9a2d-15b17643c226"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:10:03.394253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394234    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.394395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394266    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.394566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394546    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.394746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394724    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2rp\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.394827 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394765    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.394989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.394965    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.395057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.395045    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.395130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.395111    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.395200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.395150    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.395392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.395280    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b68648-ac51-409d-9a2d-15b17643c226-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:03.397341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.397287    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls" pod "f2b68648-ac51-409d-9a2d-15b17643c226" (UID: "f2b68648-ac51-409d-9a2d-15b17643c226"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:10:03.397762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.397733    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz" pod "f2b68648-ac51-409d-9a2d-15b17643c226" (UID: "f2b68648-ac51-409d-9a2d-15b17643c226"). InnerVolumeSpecName "kube-api-access-ds9pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:03.397993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.397972    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs" pod "f2b68648-ac51-409d-9a2d-15b17643c226" (UID: "f2b68648-ac51-409d-9a2d-15b17643c226"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:10:03.498639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.498586    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.498854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.498765    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2rp\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.498854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.498804    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.498994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.498965    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499029    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499094    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499132    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499172    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499210    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499232    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.499549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499365    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:03.499549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499383    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2b68648-ac51-409d-9a2d-15b17643c226-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:03.499549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499399    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9pz\" (UniqueName: \"kubernetes.io/projected/f2b68648-ac51-409d-9a2d-15b17643c226-kube-api-access-ds9pz\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:03.499843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.499819    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.500088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.500065    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.500190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.500164    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.500271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.500196    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.502975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.502949    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.503105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.503022    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.511818 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.511794    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2rp\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.511988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.511874    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.670226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.670134    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:03.975512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.972826    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:10:03.975512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.974643    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:03.975512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.974778    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:03.975512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:03.974873    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:04.011076 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.011046    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" event={"ID":"8a635544-6e59-4a28-b4d7-c3c74b5a300c","Type":"ContainerStarted","Data":"70725dd0d6d6036ef495f6b83fef417d08c24bcf3d85d5a675f012c23e884d3a"}
Nov 01 22:10:04.012652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.012623    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf" event={"ID":"f2b68648-ac51-409d-9a2d-15b17643c226","Type":"ContainerDied","Data":"437b184771c1f24d9700184953e551f5d2303a642fbc007577b8ff8d57791b88"}
Nov 01 22:10:04.012762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.012689    5845 scope.go:122] "RemoveContainer" containerID="39bd7d9d8d3b2ddf028cf0d8884c74082868472a33b64039f08d0385e4892aa2"
Nov 01 22:10:04.012762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.012638    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"
Nov 01 22:10:04.054471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.054443    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:10:04.060850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.060716    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0f9b7e-84b8587dd8-2bqsf"]
Nov 01 22:10:04.821505 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:10:04.821473    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:10:04.867089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:04.867053    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b68648-ac51-409d-9a2d-15b17643c226" path="/var/lib/kubelet/pods/f2b68648-ac51-409d-9a2d-15b17643c226/volumes"
Nov 01 22:10:05.022526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:05.022490    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" event={"ID":"8a635544-6e59-4a28-b4d7-c3c74b5a300c","Type":"ContainerStarted","Data":"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"}
Nov 01 22:10:05.085377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:05.072238    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podStartSLOduration=24.072215551 podStartE2EDuration="24.072215551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:05.05885268 +0000 UTC m=+1508.315032444" watchObservedRunningTime="2026-11-01 22:10:05.072215551 +0000 UTC m=+1508.328395316"
Nov 01 22:10:05.686642 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:05.686606    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:05.688224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:05.688182    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:05.688463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:05.688240    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:06.676339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:06.675711    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:06.676339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:06.675776    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:07.670859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:07.670820    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:07.671040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:07.670888    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:08.671465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:08.671280    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:08.671465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:08.671369    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:09.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:09.670671    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:09.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:09.670734    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:10.671370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:10.671334    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 01 22:10:10.671675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:10.671644    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 01 22:10:11.676953 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:11.676922    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:11.677516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:11.677467    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:11.678229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:11.678207    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:10:34.082637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:34.082603    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 22:10:34.083172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:34.082868    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy" containerID="cri-o://9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642" gracePeriod=30
Nov 01 22:10:34.478791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:34.478760    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 22:10:34.479011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:34.478983    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy" containerID="cri-o://c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769" gracePeriod=30
Nov 01 22:10:34.849085 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:10:34.848995    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:10:39.442733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.442705    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 22:10:39.505597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.505565    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:10:39.506597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.506568    5845 cpu_manager.go:414] "containerMap: removing container" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy"
Nov 01 22:10:39.506597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.506595    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy"
Nov 01 22:10:39.506829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.506816    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="42810644-d18a-4428-a354-4801bae499d3" containerName="istio-proxy"
Nov 01 22:10:39.516168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.514407    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.521810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.521759    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3e300c-serving-cert\""
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559187    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559276    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559378    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559500    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559605    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559731    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559902    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.559940    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.560031    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\") pod \"42810644-d18a-4428-a354-4801bae499d3\" (UID: \"42810644-d18a-4428-a354-4801bae499d3\") "
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.560527    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.561171    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp85b\" (UniqueName: \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.561218    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.561246    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.566362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.561798    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.568408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.568381    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:10:39.569733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.568989    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:10:39.569733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.569254    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.569733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.569438    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.569733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.569617    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.571959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.571935    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.575563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.575536    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:39.576367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.576343    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:10:39.576942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.576916    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc" pod "42810644-d18a-4428-a354-4801bae499d3" (UID: "42810644-d18a-4428-a354-4801bae499d3"). InnerVolumeSpecName "kube-api-access-x6vqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:39.663182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.663101    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp85b\" (UniqueName: \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.663182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.663154    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.663404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.663181    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.663404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.663239    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.664114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664085    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664117    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664130    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42810644-d18a-4428-a354-4801bae499d3-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664140    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42810644-d18a-4428-a354-4801bae499d3-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664166    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664175    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664187    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664239    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6vqc\" (UniqueName: \"kubernetes.io/projected/42810644-d18a-4428-a354-4801bae499d3-kube-api-access-x6vqc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664252    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42810644-d18a-4428-a354-4801bae499d3-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.665419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.664417    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.666436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.666416    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.666586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.666466    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.675928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.675900    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp85b\" (UniqueName: \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\") pod \"maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.813656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.813628    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 22:10:39.850552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.850512    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:10:39.851585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.851565    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy"
Nov 01 22:10:39.851653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.851587    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy"
Nov 01 22:10:39.851766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.851755    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerName="istio-proxy"
Nov 01 22:10:39.855526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.855501    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871072    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871459    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871161    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871649    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871772    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871808    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871868    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.871948    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.872282    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.872387    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\") pod \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\" (UID: \"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587\") "
Nov 01 22:10:39.873711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873185    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.873711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873246    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.873711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873365    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.873711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873613    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.873711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873669    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.874013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.873909    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.874785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.874755    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.874920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.874859    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.875384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.875153    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.876675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.876526    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdk27\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.876675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.876588    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.876675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.876667    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.879578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.876800    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.879578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.877003    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.879578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.877023    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.879578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.877038    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.879578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.877763    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:39.882974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.882950    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:10:39.883168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.883142    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:10:39.883263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.883160    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:10:39.885788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.885751    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "kube-api-access-xcsth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:39.888388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.888365    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy" pod "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" (UID: "2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:10:39.946961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.946883    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:39.978675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978652    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.978776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978695    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.978776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978737    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.978905 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978882    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdk27\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978929    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.978996    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979037    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979080    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979150    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979153    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979409    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.979508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979432    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.979508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979442    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979450    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.979508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979494    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.979798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979511    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcsth\" (UniqueName: \"kubernetes.io/projected/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587-kube-api-access-xcsth\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:39.979798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979748    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979782    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.979969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.979914    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.981637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.981615    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.981795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.981716    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.991053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.991028    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:39.991261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:39.991243    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdk27\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:40.169497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.169458    5845 generic.go:356] "Generic (PLEG): container finished" podID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" containerID="c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769" exitCode=0
Nov 01 22:10:40.170169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.169700    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" event={"ID":"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587","Type":"ContainerDied","Data":"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"}
Nov 01 22:10:40.170499 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.170463    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5" event={"ID":"2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587","Type":"ContainerDied","Data":"a1968cef79d886219ff7b1701e9e6b3d2f877ccddf2b48d839ac4acfdea3cfdb"}
Nov 01 22:10:40.170704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.170637    5845 scope.go:122] "RemoveContainer" containerID="c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"
Nov 01 22:10:40.170928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.169802    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"
Nov 01 22:10:40.175048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.174827    5845 generic.go:356] "Generic (PLEG): container finished" podID="42810644-d18a-4428-a354-4801bae499d3" containerID="9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642" exitCode=0
Nov 01 22:10:40.175048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.174897    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" event={"ID":"42810644-d18a-4428-a354-4801bae499d3","Type":"ContainerDied","Data":"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"}
Nov 01 22:10:40.175048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.174939    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx" event={"ID":"42810644-d18a-4428-a354-4801bae499d3","Type":"ContainerDied","Data":"06c9351e9febaac4c6ede58fe31a71300ea3f43390dd652512e594fe54899d43"}
Nov 01 22:10:40.175048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.174969    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"
Nov 01 22:10:40.186702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.186678    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:40.222403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.215973    5845 scope.go:122] "RemoveContainer" containerID="c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"
Nov 01 22:10:40.222403 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:10:40.217758    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769\": container with ID starting with c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769 not found: ID does not exist" containerID="c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"
Nov 01 22:10:40.222403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.217795    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769"} err="failed to get container status \"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769\": rpc error: code = NotFound desc = could not find container \"c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769\": container with ID starting with c83d87bf7d5c920e2f6e2052e6683ea3628247f5fbc3bcde414b671fa55c6769 not found: ID does not exist"
Nov 01 22:10:40.222403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.217817    5845 scope.go:122] "RemoveContainer" containerID="9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"
Nov 01 22:10:40.233403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.233369    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 22:10:40.250482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.250277    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-spwtx"]
Nov 01 22:10:40.258460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.258406    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 22:10:40.263742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.263712    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-8spl5"]
Nov 01 22:10:40.272866 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.272713    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:10:40.278007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.277977    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:10:40.306854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.306830    5845 scope.go:122] "RemoveContainer" containerID="9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"
Nov 01 22:10:40.307548 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:10:40.307473    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642\": container with ID starting with 9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642 not found: ID does not exist" containerID="9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"
Nov 01 22:10:40.307548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.307506    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642"} err="failed to get container status \"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642\": rpc error: code = NotFound desc = could not find container \"9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642\": container with ID starting with 9bc3f0b7f9d3ce9e67cc1ff87b8431aecf4936f3d6302246661e615eaff6a642 not found: ID does not exist"
Nov 01 22:10:40.609665 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.609639    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:10:40.611781 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:10:40.611755    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62e04ab_98b5_4a19_8430_f5cb71e677ff.slice/crio-22773f99efd4cc2cf34ef139f9add9eafa0437df92524942c32676cdf9bda41c WatchSource:0}: Error finding container 22773f99efd4cc2cf34ef139f9add9eafa0437df92524942c32676cdf9bda41c: Status 404 returned error can't find the container with id 22773f99efd4cc2cf34ef139f9add9eafa0437df92524942c32676cdf9bda41c
Nov 01 22:10:40.615408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.615376    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:40.615495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.615440    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:40.615495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.615468    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:40.868821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.868763    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587" path="/var/lib/kubelet/pods/2a0a37ff-7c8d-4fe8-b8d7-57ef202f4587/volumes"
Nov 01 22:10:40.869420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:40.869400    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42810644-d18a-4428-a354-4801bae499d3" path="/var/lib/kubelet/pods/42810644-d18a-4428-a354-4801bae499d3/volumes"
Nov 01 22:10:41.187256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.187166    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" event={"ID":"80b81845-ae0d-4275-a003-b475056ff7fe","Type":"ContainerStarted","Data":"49ca1d1886fd2df008b82dd9579fa8ab9c71578d8fa7d82017371521d9bf8d89"}
Nov 01 22:10:41.187256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.187224    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" event={"ID":"80b81845-ae0d-4275-a003-b475056ff7fe","Type":"ContainerStarted","Data":"de05b176a38707bd82183e1298dd305c26de227ecf2bc497f9149a4ac2514e48"}
Nov 01 22:10:41.187537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.187333    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:41.190192 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.190148    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" event={"ID":"a62e04ab-98b5-4a19-8430-f5cb71e677ff","Type":"ContainerStarted","Data":"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"}
Nov 01 22:10:41.190360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.190201    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" event={"ID":"a62e04ab-98b5-4a19-8430-f5cb71e677ff","Type":"ContainerStarted","Data":"22773f99efd4cc2cf34ef139f9add9eafa0437df92524942c32676cdf9bda41c"}
Nov 01 22:10:41.209081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.209037    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" podStartSLOduration=53.667058642 podStartE2EDuration="54.209024006s" totalImagesPullingTime="541.965364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:41.208394081 +0000 UTC m=+1544.464573844" watchObservedRunningTime="2026-11-01 22:10:41.209024006 +0000 UTC m=+1544.465203768"
Nov 01 22:10:41.237098 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:41.237053    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podStartSLOduration=15.237040848 podStartE2EDuration="15.237040848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:41.232857215 +0000 UTC m=+1544.489036979" watchObservedRunningTime="2026-11-01 22:10:41.237040848 +0000 UTC m=+1544.493220665"
Nov 01 22:10:42.187906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:42.187769    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:42.210349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:42.189774    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:42.210349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:42.189821    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:43.188019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:43.187983    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:43.188458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:43.188043    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:44.189466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:44.189398    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:44.189959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:44.189462    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:45.187384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:45.187349    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:45.187576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:45.187406    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:46.187841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:46.187803    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:46.188250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:46.187868    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:47.188080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:47.188044    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 01 22:10:47.188531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:47.188102    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 01 22:10:47.204653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:47.204625    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:10:48.192683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:48.192653    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:48.193149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:48.193011    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:48.193621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:48.193599    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:10:49.901876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:49.901844    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:10:49.902450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:49.902127    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" podUID="cb6d1743-0225-4e1e-a4d3-386b89777915" containerName="maas-api" containerID="cri-o://e4e79d4f227ba697b108fc3fe125f19d26c2d07d2261f29f7f1e841770327bd0" gracePeriod=30
Nov 01 22:10:50.349130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.349083    5845 generic.go:356] "Generic (PLEG): container finished" podID="cb6d1743-0225-4e1e-a4d3-386b89777915" containerID="e4e79d4f227ba697b108fc3fe125f19d26c2d07d2261f29f7f1e841770327bd0" exitCode=0
Nov 01 22:10:50.349130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.349141    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" event={"ID":"cb6d1743-0225-4e1e-a4d3-386b89777915","Type":"ContainerDied","Data":"e4e79d4f227ba697b108fc3fe125f19d26c2d07d2261f29f7f1e841770327bd0"}
Nov 01 22:10:50.463377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.462994    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:10:50.478072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.478026    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\") pod \"cb6d1743-0225-4e1e-a4d3-386b89777915\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") "
Nov 01 22:10:50.478262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.478176    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\") pod \"cb6d1743-0225-4e1e-a4d3-386b89777915\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") "
Nov 01 22:10:50.478407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.478387    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\") pod \"cb6d1743-0225-4e1e-a4d3-386b89777915\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") "
Nov 01 22:10:50.478518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.478483    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\" (UniqueName: \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\") pod \"cb6d1743-0225-4e1e-a4d3-386b89777915\" (UID: \"cb6d1743-0225-4e1e-a4d3-386b89777915\") "
Nov 01 22:10:50.479782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.479719    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca" pod "cb6d1743-0225-4e1e-a4d3-386b89777915" (UID: "cb6d1743-0225-4e1e-a4d3-386b89777915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:10:50.481651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.481624    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs" pod "cb6d1743-0225-4e1e-a4d3-386b89777915" (UID: "cb6d1743-0225-4e1e-a4d3-386b89777915"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:10:50.481792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.481771    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls" pod "cb6d1743-0225-4e1e-a4d3-386b89777915" (UID: "cb6d1743-0225-4e1e-a4d3-386b89777915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:10:50.481848 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.481813    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js" pod "cb6d1743-0225-4e1e-a4d3-386b89777915" (UID: "cb6d1743-0225-4e1e-a4d3-386b89777915"). InnerVolumeSpecName "kube-api-access-wc9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:10:50.504127 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.503800    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:10:50.504986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.504961    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cb6d1743-0225-4e1e-a4d3-386b89777915" containerName="maas-api"
Nov 01 22:10:50.504986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.504982    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6d1743-0225-4e1e-a4d3-386b89777915" containerName="maas-api"
Nov 01 22:10:50.505165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.505153    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6d1743-0225-4e1e-a4d3-386b89777915" containerName="maas-api"
Nov 01 22:10:50.509650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.509629    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.560428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.560382    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:10:50.580599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.580568    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.580599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.580600    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.580802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.580706    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.580843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.580806    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.580946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.580929    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.581028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581017    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.581097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581084    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgk5\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.581137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581116    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.581173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581140    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.581262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581248    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:50.581316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581266    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb6d1743-0225-4e1e-a4d3-386b89777915-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:50.581316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581278    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6d1743-0225-4e1e-a4d3-386b89777915-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:50.581316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.581287    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc9js\" (UniqueName: \"kubernetes.io/projected/cb6d1743-0225-4e1e-a4d3-386b89777915-kube-api-access-wc9js\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:10:50.683247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683164    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683201    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683255    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683335    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683479    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683669 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683588    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683669 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683605    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683687    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgk5\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683738    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683795    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683816    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.683878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.683766    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.684183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.684163    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.684370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.684347    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.685688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.685673    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.685997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.685976    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.695996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.695967    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.696591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.696573    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgk5\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\") pod \"e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") " pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:50.857387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:50.857353    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:51.164311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.164264    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:10:51.164684 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:10:51.164366    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b324fc4_13b6_4e6f_b606_5f90561a3ae6.slice/crio-42706c6710ee6c6bf9a592ce823c91cd0ee381e60266aefa87f1b6ab6de610fc WatchSource:0}: Error finding container 42706c6710ee6c6bf9a592ce823c91cd0ee381e60266aefa87f1b6ab6de610fc: Status 404 returned error can't find the container with id 42706c6710ee6c6bf9a592ce823c91cd0ee381e60266aefa87f1b6ab6de610fc
Nov 01 22:10:51.167841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.167801    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:51.167948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.167888    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:51.167948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.167931    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:10:51.363680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.363650    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"
Nov 01 22:10:51.363870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.363640    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk" event={"ID":"cb6d1743-0225-4e1e-a4d3-386b89777915","Type":"ContainerDied","Data":"e6502b2931a6e587fb8c40f58fbfdedf01e04894819b68b0addfe9dab576744d"}
Nov 01 22:10:51.363870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.363809    5845 scope.go:122] "RemoveContainer" containerID="e4e79d4f227ba697b108fc3fe125f19d26c2d07d2261f29f7f1e841770327bd0"
Nov 01 22:10:51.366710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.366670    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" event={"ID":"7b324fc4-13b6-4e6f-b606-5f90561a3ae6","Type":"ContainerStarted","Data":"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"}
Nov 01 22:10:51.366821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.366747    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" event={"ID":"7b324fc4-13b6-4e6f-b606-5f90561a3ae6","Type":"ContainerStarted","Data":"42706c6710ee6c6bf9a592ce823c91cd0ee381e60266aefa87f1b6ab6de610fc"}
Nov 01 22:10:51.385718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.385687    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:10:51.390338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.390312    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e2a91d-6545dd95fd-z5sxk"]
Nov 01 22:10:51.416191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.416141    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podStartSLOduration=25.416128622 podStartE2EDuration="25.416128622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:10:51.411863445 +0000 UTC m=+1554.668043209" watchObservedRunningTime="2026-11-01 22:10:51.416128622 +0000 UTC m=+1554.672308384"
Nov 01 22:10:51.858072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:51.858042    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:52.869772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:52.869733    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:52.870338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:52.869796    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:52.883995 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:52.883961    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6d1743-0225-4e1e-a4d3-386b89777915" path="/var/lib/kubelet/pods/cb6d1743-0225-4e1e-a4d3-386b89777915/volumes"
Nov 01 22:10:53.869742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:53.869701    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:53.869935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:53.869767    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:54.873891 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:54.873846    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:54.874549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:54.874507    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:55.875318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:55.870653    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:55.875318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:55.870718    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:56.869676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:56.869638    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:56.869877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:56.869695    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:57.857718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:57.857679    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:10:57.858147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:57.857735    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 01 22:10:58.908643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:58.908614    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:58.909338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:58.909277    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:10:58.910800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:10:58.910766    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:11:04.825002 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:11:04.824965    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:11:34.818708 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:11:34.818664    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:12:05.143918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:05.143865    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:05.437097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:05.379753    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:05.619405 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:12:05.619279    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:12:07.217846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:07.217798    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:07.402528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:07.402487    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:08.362569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:08.362479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:12:08.453464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:08.362718    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 01 22:12:11.128075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.128009    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.203366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.167625    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:12:11.203366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.168105    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:12:11.203366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.168398    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:12:11.203366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.173924    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 01 22:12:11.203366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.174421    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.518699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.456622    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.518699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.483477    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.518699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.483781    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.668253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.668206    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:12:11.679693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.679656    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 01 22:12:11.809829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.809741    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.973681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.838898    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:11.973681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.838950    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:11.973681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:11.891189    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.122:50052\" within 1s: context deadline exceeded"
Nov 01 22:12:11.973681 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:12:11.891720    5845 logging.go:55] [core] [Channel #2866 SubChannel #2881]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.122:50052", ServerName: "10.134.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.122:50052: operation was canceled"
Nov 01 22:12:12.181020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.180945    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:12.181135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.181013    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:12.427459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.427421    5845 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dsdmp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:12.558630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.427483    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:12.704478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.704433    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.714455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.714428    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dfdb11\"/\"openshift-service-ca.crt\""
Nov 01 22:12:12.714613 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.714520    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dfdb11\"/\"default-dockercfg-2l2b7\""
Nov 01 22:12:12.714613 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.714570    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dfdb11\"/\"test-model-dfdb11-kserve-self-signed-certs\""
Nov 01 22:12:12.714613 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.714609    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dfdb11\"/\"kube-root-ca.crt\""
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721317    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721530    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721597    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721721    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721752    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxfq\" (UniqueName: \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.747954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.721846    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.810342    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" start-of-body=
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.810390    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.822668    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.822902    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.822943    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxfq\" (UniqueName: \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.823054    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.823143    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:12.900475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:12.823364    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:13.450483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:13.426407    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:13.450483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:13.445946    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:13.450483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:13.445995    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:14.219437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:14.219401    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:14.291805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:14.219460    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:15.027117 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:15.027076    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:15.200724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:15.200692    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:15.348261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:15.348190    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:15.631921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:15.631850    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:15.884622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:15.884550    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxfq\" (UniqueName: \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:16.077691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:16.077659    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\") pod \"test-model-dfdb11-kserve-6978b659d7-npnmc\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") " pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:16.160975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:16.160890    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:12:19.219709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:18.285880    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:19.487547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:19.487457    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:20.026420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:20.026389    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:20.026596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:20.026447    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:21.483453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:21.483408    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:21.483952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:21.483469    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:21.643204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:21.643089    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 01 22:12:21.643204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:21.643156    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded"
Nov 01 22:12:29.723421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:29.723384    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:29.807259 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:29.723445    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:30.689235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:30.689198    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 01 22:12:31.115454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:31.115414    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:31.116197 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:31.115494    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:31.442960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:31.442850    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:31.442960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:31.442931    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:31.588787 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:31.588744    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:32.705433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:32.704595    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 22:12:32.741514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:32.737107    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 22:12:34.021879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.021820    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:34.069048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.031058    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:12:34.069048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.054503    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:34.182217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.072904    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.203688    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.515000    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.515945    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:34.516483    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:35.574425    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:36.255453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:35.574496    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:38.481844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:37.803953    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.105:50052: i/o timeout"
Nov 01 22:12:38.481844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:37.878471    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:41.394086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:41.393941    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:41.535020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:41.534941    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.141143738s: [/var/lib/containers/storage/overlay/b3c53a72b0d68e556698c10171b57ef800657bcf702a42bda778727b9703a390/diff ]; will not log again for this container unless duration exceeds 2s
Nov 01 22:12:41.578604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:41.578563    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.078528514s: [/var/lib/containers/storage/overlay/a3d19c19f98679189711dd0331ac6045d739ca10d14d3d5d53a1cfe59a786110/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc_c2b65487-1cd9-4053-bb63-f50c185beb79/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:12:41.584576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:41.584547    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.071880775s: [/var/lib/containers/storage/overlay/993ac6335d91af32e36225526013dbf1b67779e3fe34083d6ebdb3b10f084eb3/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-75d8948445-6nwm2_533a4e94-3fb6-4bb4-b248-0a256638d509/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:12:42.447767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:42.420695    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:42.447767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:42.432400    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:42.524457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:42.524424    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:42.524618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:42.524525    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.383108 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:12:43.383058    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:12:43.399835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.399813    5845 trace.go:236] Trace[297277093]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" (01-Nov-2026 22:12:33.400) (total time: 9999ms):
Nov 01 22:12:43.399835 ip-10-0-135-146 kubenswrapper[5845]: Trace[297277093]: [9.999070563s] [9.999070563s] END
Nov 01 22:12:43.438587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.421322    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.445518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.445463    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:43.483978 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:12:43.483945    5845 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.439s"
Nov 01 22:12:43.563870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.563824    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.586426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.586380    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.586426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.586388    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:12:43.586664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.586394    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.589577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.589550    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.770689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.770649    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:43.770689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.770662    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.770728    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.787818    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.787879    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.795052    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.825870    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.833320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.825899    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.884354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.884289    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.920336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.920285    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:43.961742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.961711    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:12:43.961932 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.961834    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:12:43.961932 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:43.961892    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:12:44.538411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:44.538379    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:12:44.538822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:44.538430    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:12:45.621490 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:12:45.615792    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f546dd9_4d51_45f6_aea1_62060513f622.slice/crio-4f56c935367d12cee7f31e9ed7992a18b2fee1449d8022544b30853b3c7e2286 WatchSource:0}: Error finding container 4f56c935367d12cee7f31e9ed7992a18b2fee1449d8022544b30853b3c7e2286: Status 404 returned error can't find the container with id 4f56c935367d12cee7f31e9ed7992a18b2fee1449d8022544b30853b3c7e2286
Nov 01 22:12:46.224157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:46.224124    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" event={"ID":"3f546dd9-4d51-45f6-aea1-62060513f622","Type":"ContainerStarted","Data":"4f56c935367d12cee7f31e9ed7992a18b2fee1449d8022544b30853b3c7e2286"}
Nov 01 22:12:47.236960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:47.236916    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" event={"ID":"3f546dd9-4d51-45f6-aea1-62060513f622","Type":"ContainerStarted","Data":"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"}
Nov 01 22:12:47.818965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:47.818931    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:48.985230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:48.985204    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:48.985714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:48.985252    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:48.985714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:48.985364    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:12:48.985988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:48.985963    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:48.986036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:48.986011    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:49.012205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.012153    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" podStartSLOduration=43.012128245 podStartE2EDuration="43.012128245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:12:47.267098257 +0000 UTC m=+1670.523278022" watchObservedRunningTime="2026-11-01 22:12:49.012128245 +0000 UTC m=+1672.268308007"
Nov 01 22:12:49.255479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.255445    5845 generic.go:356] "Generic (PLEG): container finished" podID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerID="a86a5daa0974e76fd6554636de7018179747532175030083000f29c213ecbdf3" exitCode=0
Nov 01 22:12:49.255679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.255504    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerDied","Data":"a86a5daa0974e76fd6554636de7018179747532175030083000f29c213ecbdf3"}
Nov 01 22:12:49.257583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.257561    5845 scope.go:122] "RemoveContainer" containerID="a86a5daa0974e76fd6554636de7018179747532175030083000f29c213ecbdf3"
Nov 01 22:12:49.259488 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.259461    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:12:49.259605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.259529    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:12:49.259605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:49.259569    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:12:50.266091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:50.266054    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerStarted","Data":"7e1687925f3863aa39d235fa08d56f3af48e294803cc81f90711dbcd99b57415"}
Nov 01 22:12:51.008061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:51.008026    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:12:51.008265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:51.008158    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:51.008265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:51.008219    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:52.008224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:52.008189    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:52.008623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:52.008248    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:53.017285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:53.017250    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:53.020437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:53.017972    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:54.009081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:54.009048    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:54.009249 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:54.009096    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:55.008426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:55.008390    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:55.008805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:55.008472    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:56.008412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:56.008377    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 01 22:12:56.008628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:56.008431    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 01 22:12:56.818720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:56.818687    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:56.832053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:56.832025    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:12:57.013791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:57.013761    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:12:57.014783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:57.014275    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:12:57.014874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:57.014853    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:12:57.408736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:12:57.408702    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:13:04.846812 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:13:04.846776    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:13:33.855549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:33.855513    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:13:34.843195 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:13:34.843158    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:13:38.856556 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:38.856440    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing" containerID="cri-o://3afb26c5936811d30fb4ed3327953b6e2a2a8c3e204ce44fe0e495c23b85da93" gracePeriod=25
Nov 01 22:13:40.638938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:40.638903    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:13:49.493080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:49.493040    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 22:13:49.493697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:49.493243    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy" containerID="cri-o://192beb8743743bb255e05521ff5a2ba51950f2a43d8940fd4526a969042ace28" gracePeriod=30
Nov 01 22:13:49.498286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:49.498255    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 22:13:49.498528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:49.498501    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" containerID="cri-o://6c45d3b91641321fa996b78023ed33c471107aa84991b4aabbb6be5da942752e" gracePeriod=30
Nov 01 22:13:50.638537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:50.638492    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:13:55.073348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.072504    5845 generic.go:356] "Generic (PLEG): container finished" podID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerID="192beb8743743bb255e05521ff5a2ba51950f2a43d8940fd4526a969042ace28" exitCode=0
Nov 01 22:13:55.073348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.072621    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" event={"ID":"9fbd6256-a14f-4146-9993-6643d96ddee8","Type":"ContainerDied","Data":"192beb8743743bb255e05521ff5a2ba51950f2a43d8940fd4526a969042ace28"}
Nov 01 22:13:55.074444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.074425    5845 generic.go:356] "Generic (PLEG): container finished" podID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerID="6c45d3b91641321fa996b78023ed33c471107aa84991b4aabbb6be5da942752e" exitCode=0
Nov 01 22:13:55.075018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.074692    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" event={"ID":"b1c93c32-5378-4a0d-97e9-dda679b6aad0","Type":"ContainerDied","Data":"6c45d3b91641321fa996b78023ed33c471107aa84991b4aabbb6be5da942752e"}
Nov 01 22:13:55.339949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.338509    5845 patch_prober.go:29] interesting pod/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 01 22:13:55.339949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.338613    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 01 22:13:55.523908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.523875    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 22:13:55.570777 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.570747    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:13:55.572530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.572499    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy"
Nov 01 22:13:55.572706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.572691    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy"
Nov 01 22:13:55.573035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.573015    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" containerName="istio-proxy"
Nov 01 22:13:55.593555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.584865    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.613557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.613526    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.623996    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.629892    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.629984    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630032    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630081    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630129    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxgl\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630167    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630290    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.635410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.630406    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.646180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.646152    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 22:13:55.693332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.692232    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:13:55.696327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.694806    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy"
Nov 01 22:13:55.696327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.694834    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy"
Nov 01 22:13:55.696327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.695142    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" containerName="istio-proxy"
Nov 01 22:13:55.704318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.699823    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.724373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.723704    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:13:55.735342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.735269    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736204    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736362    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736547    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736620    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736656    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736783    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736820    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736893    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736920    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.736988    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737044    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.737284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737142    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737647    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737694    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737727    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\") pod \"9fbd6256-a14f-4146-9993-6643d96ddee8\" (UID: \"9fbd6256-a14f-4146-9993-6643d96ddee8\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737808    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.737848    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\") pod \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\" (UID: \"b1c93c32-5378-4a0d-97e9-dda679b6aad0\") "
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.738245    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.738970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.738350    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739074    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.739324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739174    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739203    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqxh\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.739482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739438    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.739482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739474    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739642    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739706    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739828 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739772    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.739885 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739835    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.739964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.739910    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxgl\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.740026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740011    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.741010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740077    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.741010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740104    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.741010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740620    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.741010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740747    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.741010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.740804    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.741322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.741118    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:13:55.741708 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.741686    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.742656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.742565    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.749340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.743988    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:13:55.754726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.754498    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.755524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.755500    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.756832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.756747    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.769481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.758484    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.769481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.758715    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.769481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.759025    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:55.810655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.790483    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.810655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.810462    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.877786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.854563    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.877786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.856481    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.877786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.857583    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.877786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.857775    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.893954    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894008    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894063    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894149    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894210    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894412    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894515    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqxh\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894655    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894858    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.894987    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895007    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895024    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895038    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895053    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd6256-a14f-4146-9993-6643d96ddee8-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895067    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895080    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895094    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895108    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895122    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895137    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.895734    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.896218    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.897557    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.898383    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:55.920386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:55.898782    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.019813    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.033422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.033577    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.040332    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.040358    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1c93c32-5378-4a0d-97e9-dda679b6aad0-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.043476    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:13:56.044904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.044001    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqxh\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.071733 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.054357    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.077120    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh" pod "b1c93c32-5378-4a0d-97e9-dda679b6aad0" (UID: "b1c93c32-5378-4a0d-97e9-dda679b6aad0"). InnerVolumeSpecName "kube-api-access-s7bjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.077612    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.077693    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.078078    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.078642    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.079102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.078898    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z" pod "9fbd6256-a14f-4146-9993-6643d96ddee8" (UID: "9fbd6256-a14f-4146-9993-6643d96ddee8"). InnerVolumeSpecName "kube-api-access-g5q8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:13:56.089700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.089395    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxgl\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:56.106391 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.106122    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:56.108364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.107609    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:56.108364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.107989    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\") pod \"e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") " pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:56.139462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.139386    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"
Nov 01 22:13:56.139462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.139397    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5" event={"ID":"9fbd6256-a14f-4146-9993-6643d96ddee8","Type":"ContainerDied","Data":"ab2bfe09507c171bae36b47b40acb707ca1ba99605b4a83e76db46f8779d34f1"}
Nov 01 22:13:56.139694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.139502    5845 scope.go:122] "RemoveContainer" containerID="192beb8743743bb255e05521ff5a2ba51950f2a43d8940fd4526a969042ace28"
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.141649    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665" event={"ID":"b1c93c32-5378-4a0d-97e9-dda679b6aad0","Type":"ContainerDied","Data":"6f1fe807ada5e048043d843228d95feaa9f568f0fd4d0c4dd95dde8b083d06e5"}
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.141648    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.142313    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.142343    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bjh\" (UniqueName: \"kubernetes.io/projected/b1c93c32-5378-4a0d-97e9-dda679b6aad0-kube-api-access-s7bjh\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.142361    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.142373    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fbd6256-a14f-4146-9993-6643d96ddee8-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.147134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.142388    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/9fbd6256-a14f-4146-9993-6643d96ddee8-kube-api-access-g5q8z\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:56.160798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.160776    5845 scope.go:122] "RemoveContainer" containerID="6c45d3b91641321fa996b78023ed33c471107aa84991b4aabbb6be5da942752e"
Nov 01 22:13:56.200662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.200629    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 22:13:56.203165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.203136    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-kjqj5"]
Nov 01 22:13:56.208380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.207897    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 22:13:56.214899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.213864    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f033b1-openshift-default-5b679ffdf9-s4665"]
Nov 01 22:13:56.271620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.266344    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:56.415726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.415086    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:13:56.415726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.415381    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy" containerID="cri-o://9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0" gracePeriod=30
Nov 01 22:13:56.421497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.421424    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:13:56.421868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.421655    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy" containerID="cri-o://7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38" gracePeriod=30
Nov 01 22:13:56.498817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.498787    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:13:56.499753 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:13:56.499728    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda180a956_8b70_4fba_855b_eacd78570803.slice/crio-7411cdb6de3ae8715e94942ce547e51bf45ed7a59ae699ace1b227470c1e9477 WatchSource:0}: Error finding container 7411cdb6de3ae8715e94942ce547e51bf45ed7a59ae699ace1b227470c1e9477: Status 404 returned error can't find the container with id 7411cdb6de3ae8715e94942ce547e51bf45ed7a59ae699ace1b227470c1e9477
Nov 01 22:13:56.530407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.530204    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.530407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.530337    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.530407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.530394    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.786965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.786932    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:13:56.803592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.803205    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.803592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.803326    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.803592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.803374    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:13:56.881210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.880790    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbd6256-a14f-4146-9993-6643d96ddee8" path="/var/lib/kubelet/pods/9fbd6256-a14f-4146-9993-6643d96ddee8/volumes"
Nov 01 22:13:56.882355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:56.882026    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c93c32-5378-4a0d-97e9-dda679b6aad0" path="/var/lib/kubelet/pods/b1c93c32-5378-4a0d-97e9-dda679b6aad0/volumes"
Nov 01 22:13:57.168250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.168211    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" event={"ID":"c70e5a04-ae17-4f6b-bfa7-4908166d4e33","Type":"ContainerStarted","Data":"656b2b3ab08fed1bccd17a0636f91333f87f68cc3e5a33c964db4f50fe8c355b"}
Nov 01 22:13:57.168853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.168825    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" event={"ID":"c70e5a04-ae17-4f6b-bfa7-4908166d4e33","Type":"ContainerStarted","Data":"8cbc476cb05e58354f2b39b64367ca9ea310004ad9043a6b41d091a1b3374583"}
Nov 01 22:13:57.172079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.172055    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" event={"ID":"a180a956-8b70-4fba-855b-eacd78570803","Type":"ContainerStarted","Data":"776a37f3bb38da1e6ebf37dd76643e58ec42b11fc5265fea8861593ae214c34a"}
Nov 01 22:13:57.172376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.172354    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" event={"ID":"a180a956-8b70-4fba-855b-eacd78570803","Type":"ContainerStarted","Data":"7411cdb6de3ae8715e94942ce547e51bf45ed7a59ae699ace1b227470c1e9477"}
Nov 01 22:13:57.200804 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.200752    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podStartSLOduration=86.20073441 podStartE2EDuration="1m26.20073441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:57.195413811 +0000 UTC m=+1740.451593575" watchObservedRunningTime="2026-11-01 22:13:57.20073441 +0000 UTC m=+1740.456914175"
Nov 01 22:13:57.225826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.223393    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podStartSLOduration=86.223377725 podStartE2EDuration="1m26.223377725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:13:57.219924966 +0000 UTC m=+1740.476104742" watchObservedRunningTime="2026-11-01 22:13:57.223377725 +0000 UTC m=+1740.479557489"
Nov 01 22:13:57.267124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.267088    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:13:57.268648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.268617    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:13:57.268800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:57.268671    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:13:58.120321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:58.117456    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:13:58.120321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:58.117771    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:13:58.120321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:58.117828    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:13:58.307242 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:58.307195    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:13:58.307932 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:58.307281    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:13:59.114406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.114130    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:13:59.114406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.114179    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:13:59.233717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.233204    5845 generic.go:356] "Generic (PLEG): container finished" podID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerID="3afb26c5936811d30fb4ed3327953b6e2a2a8c3e204ce44fe0e495c23b85da93" exitCode=0
Nov 01 22:13:59.233717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.233282    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" event={"ID":"4cff68f7-cebc-4677-9548-fd42f95cbcf0","Type":"ContainerDied","Data":"3afb26c5936811d30fb4ed3327953b6e2a2a8c3e204ce44fe0e495c23b85da93"}
Nov 01 22:13:59.303713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.303190    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:13:59.303713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.303272    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:13:59.372728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.372696    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:13:59.418975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.418895    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\") pod \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") "
Nov 01 22:13:59.419274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.419255    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\" (UniqueName: \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\") pod \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\" (UID: \"4cff68f7-cebc-4677-9548-fd42f95cbcf0\") "
Nov 01 22:13:59.419450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.419423    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume" pod "4cff68f7-cebc-4677-9548-fd42f95cbcf0" (UID: "4cff68f7-cebc-4677-9548-fd42f95cbcf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:13:59.420462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.420439    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cff68f7-cebc-4677-9548-fd42f95cbcf0-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:13:59.428345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.425219    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr" pod "4cff68f7-cebc-4677-9548-fd42f95cbcf0" (UID: "4cff68f7-cebc-4677-9548-fd42f95cbcf0"). InnerVolumeSpecName "kube-api-access-kgbjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:13:59.522368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:13:59.522289    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbjr\" (UniqueName: \"kubernetes.io/projected/4cff68f7-cebc-4677-9548-fd42f95cbcf0-kube-api-access-kgbjr\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:00.121922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.121889    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:14:00.122650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.122615    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:14:00.243231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.243198    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"
Nov 01 22:14:00.243440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.243231    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz" event={"ID":"4cff68f7-cebc-4677-9548-fd42f95cbcf0","Type":"ContainerDied","Data":"4f333eff52ca95f321adaf9f118d4bd2fd24d67cb9406efae3e7a8feb3e8f331"}
Nov 01 22:14:00.243440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.243289    5845 scope.go:122] "RemoveContainer" containerID="3afb26c5936811d30fb4ed3327953b6e2a2a8c3e204ce44fe0e495c23b85da93"
Nov 01 22:14:00.290331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.286837    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:14:00.290331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.286910    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:14:00.323279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.323244    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:14:00.327487 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.327467    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edd32a-76686b7d6-tggwz"]
Nov 01 22:14:00.900783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:00.900748    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" path="/var/lib/kubelet/pods/4cff68f7-cebc-4677-9548-fd42f95cbcf0/volumes"
Nov 01 22:14:01.107890 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:01.107860    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:14:01.108111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:01.108089    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:14:01.290323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:01.271110    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:14:01.290323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:01.271170    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:14:02.127285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.127245    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:14:02.127718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.127333    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:14:02.295412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.282693    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:14:02.295412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.282748    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:14:02.312292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.312259    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:14:02.316408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.316386    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:14:02.318181 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.318156    5845 generic.go:356] "Generic (PLEG): container finished" podID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerID="9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0" exitCode=0
Nov 01 22:14:02.318321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.318283    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" event={"ID":"b4761bc6-7eb1-44bd-8eb6-cd7a07396377","Type":"ContainerDied","Data":"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"}
Nov 01 22:14:02.318433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.318356    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht" event={"ID":"b4761bc6-7eb1-44bd-8eb6-cd7a07396377","Type":"ContainerDied","Data":"93d958c1ba2ca2612e722971da4735056b657f0c9a064984dd1a18b478a22574"}
Nov 01 22:14:02.318433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.318380    5845 scope.go:122] "RemoveContainer" containerID="9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"
Nov 01 22:14:02.318554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.318460    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"
Nov 01 22:14:02.320071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.320047    5845 generic.go:356] "Generic (PLEG): container finished" podID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerID="7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38" exitCode=0
Nov 01 22:14:02.320170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.320087    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"
Nov 01 22:14:02.320208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.320173    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" event={"ID":"ca4683d5-2dca-4f63-a9ab-2bd314e8051d","Type":"ContainerDied","Data":"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"}
Nov 01 22:14:02.320254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.320216    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82" event={"ID":"ca4683d5-2dca-4f63-a9ab-2bd314e8051d","Type":"ContainerDied","Data":"125622bcd3acb42d5d593b97398a807c3a056705f66303a2cec0c62b237478b9"}
Nov 01 22:14:02.331265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.331247    5845 scope.go:122] "RemoveContainer" containerID="9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"
Nov 01 22:14:02.331567 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:02.331547    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0\": container with ID starting with 9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0 not found: ID does not exist" containerID="9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"
Nov 01 22:14:02.331628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.331577    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0"} err="failed to get container status \"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0\": rpc error: code = NotFound desc = could not find container \"9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0\": container with ID starting with 9dc56bb21199ebd99a0b0bb31c1f9a0c0406e67e05bd16d12b753d91057a67c0 not found: ID does not exist"
Nov 01 22:14:02.331628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.331593    5845 scope.go:122] "RemoveContainer" containerID="7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"
Nov 01 22:14:02.337685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337661    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.337778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337706    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.337778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337764    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.337881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337814    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.337881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337846    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.337881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337871    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.338042 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337909    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338042 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337952    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.338042 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.337987    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.338190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338041    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.338190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338150    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338201    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338234    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338278    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.338473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338330    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338421    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\") pod \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\" (UID: \"ca4683d5-2dca-4f63-a9ab-2bd314e8051d\") "
Nov 01 22:14:02.338473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338424    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.338617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338538    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.338719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338702    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.339289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338889    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\") pod \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\" (UID: \"b4761bc6-7eb1-44bd-8eb6-cd7a07396377\") "
Nov 01 22:14:02.339289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338711    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.339289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338726    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.339289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.338906    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:02.340266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.340241    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341461    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341502    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341518    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.340725    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341532    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.340743    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.340911    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "kube-api-access-hzpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.340990    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341041    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341244    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:02.341617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.341507    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:14:02.342291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.342268    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:02.342809 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.342539    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.342809 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.342550    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:14:02.342963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.342810    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy" pod "ca4683d5-2dca-4f63-a9ab-2bd314e8051d" (UID: "ca4683d5-2dca-4f63-a9ab-2bd314e8051d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.343597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.343573    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "kube-api-access-jmws6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:02.344026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.344007    5845 scope.go:122] "RemoveContainer" containerID="7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"
Nov 01 22:14:02.344105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.344086    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy" pod "b4761bc6-7eb1-44bd-8eb6-cd7a07396377" (UID: "b4761bc6-7eb1-44bd-8eb6-cd7a07396377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:02.423348 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:02.415844    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38\": container with ID starting with 7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38 not found: ID does not exist" containerID="7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"
Nov 01 22:14:02.423348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.418541    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38"} err="failed to get container status \"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38\": rpc error: code = NotFound desc = could not find container \"7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38\": container with ID starting with 7034e06b4faf6a1c3cfb94fcd73085cf20eec0e3fb94b318037294cf680aaf38 not found: ID does not exist"
Nov 01 22:14:02.433628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.433602    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:14:02.434732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434710    5845 cpu_manager.go:414] "containerMap: removing container" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing"
Nov 01 22:14:02.434732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434732    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434754    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434759    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434790    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434795    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434930    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cff68f7-cebc-4677-9548-fd42f95cbcf0" containerName="payload-processing"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434940    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" containerName="istio-proxy"
Nov 01 22:14:02.435694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.434951    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" containerName="istio-proxy"
Nov 01 22:14:02.440611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.440561    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:14:02.441976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.440677    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444001    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444029    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444046    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444060    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444074    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpdg\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-kube-api-access-hzpdg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444089    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444171    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444189    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444203    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4683d5-2dca-4f63-a9ab-2bd314e8051d-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444217    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444231    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmws6\" (UniqueName: \"kubernetes.io/projected/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-kube-api-access-jmws6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444245    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.444259    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4761bc6-7eb1-44bd-8eb6-cd7a07396377-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:02.483715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.457663    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7e2df-openshift-default-dockercfg-tmrhj\""
Nov 01 22:14:02.573602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.553519    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqdmb\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.574764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.574732    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.595793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.595744    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.597221 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.597057    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.597568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.597544    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.597744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.597726    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.598150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.598016    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.598150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.598105    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.598324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.598159    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.697555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.697480    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700101    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700404    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqdmb\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700470    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700522    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700554    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700570    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700702    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700765    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700835    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.701414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.700899    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.702566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.701959    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.702566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.702204    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.702566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.702261    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.702902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.702866    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.704363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.704346    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.705626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.704889    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.707640 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.707617    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-mrhht"]
Nov 01 22:14:02.714455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.712434    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:14:02.717223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.717155    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.717416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.717397    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqdmb\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\") pod \"e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") " pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.721022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.721000    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-edd32a-openshift-default-7cf5f49d65-kgh82"]
Nov 01 22:14:02.827643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.827555    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:02.917449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.917414    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4761bc6-7eb1-44bd-8eb6-cd7a07396377" path="/var/lib/kubelet/pods/b4761bc6-7eb1-44bd-8eb6-cd7a07396377/volumes"
Nov 01 22:14:02.918022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:02.918005    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4683d5-2dca-4f63-a9ab-2bd314e8051d" path="/var/lib/kubelet/pods/ca4683d5-2dca-4f63-a9ab-2bd314e8051d/volumes"
Nov 01 22:14:03.108158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.107644    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 01 22:14:03.108158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.107704    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 01 22:14:03.206928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.206901    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:14:03.207347 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:14:03.207058    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f10c1d2_c233_4174_939b_676ec58c594d.slice/crio-cc331a195834ae8b4fe255ccf24cc38bce285333dfd4d821b609ba272c4c29f7 WatchSource:0}: Error finding container cc331a195834ae8b4fe255ccf24cc38bce285333dfd4d821b609ba272c4c29f7: Status 404 returned error can't find the container with id cc331a195834ae8b4fe255ccf24cc38bce285333dfd4d821b609ba272c4c29f7
Nov 01 22:14:03.210857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.210826    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:03.210930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.210893    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:03.210930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.210925    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:03.267036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.267012    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 01 22:14:03.267193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.267063    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 01 22:14:03.344616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.344576    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerStarted","Data":"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"}
Nov 01 22:14:03.344802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.344645    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerStarted","Data":"cc331a195834ae8b4fe255ccf24cc38bce285333dfd4d821b609ba272c4c29f7"}
Nov 01 22:14:03.392374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.376397    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podStartSLOduration=184.376377145 podStartE2EDuration="3m4.376377145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:03.369847919 +0000 UTC m=+1746.626027683" watchObservedRunningTime="2026-11-01 22:14:03.376377145 +0000 UTC m=+1746.632556909"
Nov 01 22:14:03.830097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:03.830051    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:04.112235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.112165    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:14:04.112659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.112643    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:14:04.113260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.113240    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:14:04.271505 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.271471    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:14:04.272001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.271966    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:14:04.272660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.272641    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:14:04.825085 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:04.825048    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:14:04.834112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:04.834089    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:05.368478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:05.368445    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:05.369596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:05.369572    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:14:17.353370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:17.353329    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:14:17.369206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:17.369174    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:14:20.995004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:20.994965    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:14:20.995622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:20.995205    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" podUID="347a9163-6321-49db-a829-fc2b8cfa211e" containerName="maas-api" containerID="cri-o://84b06194d5774701dc959fff719909c28fce1f2779faa979435f6d3a71ca7958" gracePeriod=30
Nov 01 22:14:21.586386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.581698    5845 generic.go:356] "Generic (PLEG): container finished" podID="347a9163-6321-49db-a829-fc2b8cfa211e" containerID="84b06194d5774701dc959fff719909c28fce1f2779faa979435f6d3a71ca7958" exitCode=0
Nov 01 22:14:21.586386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.581995    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" event={"ID":"347a9163-6321-49db-a829-fc2b8cfa211e","Type":"ContainerDied","Data":"84b06194d5774701dc959fff719909c28fce1f2779faa979435f6d3a71ca7958"}
Nov 01 22:14:21.711756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.711725    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.832945    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\") pod \"347a9163-6321-49db-a829-fc2b8cfa211e\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") "
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.833059    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\" (UniqueName: \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\") pod \"347a9163-6321-49db-a829-fc2b8cfa211e\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") "
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.833188    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\") pod \"347a9163-6321-49db-a829-fc2b8cfa211e\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") "
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.833249    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\") pod \"347a9163-6321-49db-a829-fc2b8cfa211e\" (UID: \"347a9163-6321-49db-a829-fc2b8cfa211e\") "
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.838249    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca" pod "347a9163-6321-49db-a829-fc2b8cfa211e" (UID: "347a9163-6321-49db-a829-fc2b8cfa211e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:21.847531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.840215    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn" pod "347a9163-6321-49db-a829-fc2b8cfa211e" (UID: "347a9163-6321-49db-a829-fc2b8cfa211e"). InnerVolumeSpecName "kube-api-access-4qnwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:21.849310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.849234    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs" pod "347a9163-6321-49db-a829-fc2b8cfa211e" (UID: "347a9163-6321-49db-a829-fc2b8cfa211e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:14:21.870389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.853900    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls" pod "347a9163-6321-49db-a829-fc2b8cfa211e" (UID: "347a9163-6321-49db-a829-fc2b8cfa211e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:14:21.938113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.938076    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:21.938384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.938348    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qnwn\" (UniqueName: \"kubernetes.io/projected/347a9163-6321-49db-a829-fc2b8cfa211e-kube-api-access-4qnwn\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:21.938384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.938377    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347a9163-6321-49db-a829-fc2b8cfa211e-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:21.938562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:21.938391    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347a9163-6321-49db-a829-fc2b8cfa211e-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:22.354099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.354056    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing" containerID="cri-o://904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618" gracePeriod=25
Nov 01 22:14:22.370392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.370328    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing" containerID="cri-o://58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1" gracePeriod=25
Nov 01 22:14:22.593855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.593818    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p" event={"ID":"347a9163-6321-49db-a829-fc2b8cfa211e","Type":"ContainerDied","Data":"d156e420213a760accf08b5a7369bb59a99c1222ad3ca6f6725ba19cee2590f6"}
Nov 01 22:14:22.594043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.593884    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"
Nov 01 22:14:22.594043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.593908    5845 scope.go:122] "RemoveContainer" containerID="84b06194d5774701dc959fff719909c28fce1f2779faa979435f6d3a71ca7958"
Nov 01 22:14:22.624032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.624004    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:14:22.629924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.629898    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3577a0-6db6d67764-lhr7p"]
Nov 01 22:14:22.865259 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:22.865167    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347a9163-6321-49db-a829-fc2b8cfa211e" path="/var/lib/kubelet/pods/347a9163-6321-49db-a829-fc2b8cfa211e/volumes"
Nov 01 22:14:29.205433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:29.205395    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:14:29.205952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:29.205660    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy" containerID="cri-o://41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7" gracePeriod=30
Nov 01 22:14:30.328031 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.327997    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:14:30.329158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.329136    5845 cpu_manager.go:414] "containerMap: removing container" podUID="347a9163-6321-49db-a829-fc2b8cfa211e" containerName="maas-api"
Nov 01 22:14:30.329158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.329158    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347a9163-6321-49db-a829-fc2b8cfa211e" containerName="maas-api"
Nov 01 22:14:30.329847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.329427    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="347a9163-6321-49db-a829-fc2b8cfa211e" containerName="maas-api"
Nov 01 22:14:30.335224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.335205    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.348200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.345240    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d6e25f-openshift-default-dockercfg-t9xvc\""
Nov 01 22:14:30.348200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.347476    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.535711    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.535784    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.535904    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.535992    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.536088    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.536111    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mrp\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.536255    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.536309    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.563348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.536358    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.640663    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.640727    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.640823    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.640901    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.640987    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.641011    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mrp\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.641143    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.641181    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.641226    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.641793    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.650958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.642905    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.706319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.660038    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.706319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.661458    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.722751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.716508    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.722751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.716558    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.743352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.725868    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.743352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.730136    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.774111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.774085    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mrp\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\") pod \"e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") " pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:30.958167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:30.958094    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:31.252845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.252810    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:14:31.253826 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:14:31.253785    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9871b3c6_0309_406a_b02b_8696a0daf755.slice/crio-d348d20d348c842e8027a83e1eeb07b24a106d78d438896d1207738bcb5a5f81 WatchSource:0}: Error finding container d348d20d348c842e8027a83e1eeb07b24a106d78d438896d1207738bcb5a5f81: Status 404 returned error can't find the container with id d348d20d348c842e8027a83e1eeb07b24a106d78d438896d1207738bcb5a5f81
Nov 01 22:14:31.257234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.257199    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:31.257374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.257288    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:31.257374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.257348    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:14:31.708068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.707983    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:14:31.708479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.708063    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:14:31.798670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.798635    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerStarted","Data":"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"}
Nov 01 22:14:31.798856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.798733    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerStarted","Data":"d348d20d348c842e8027a83e1eeb07b24a106d78d438896d1207738bcb5a5f81"}
Nov 01 22:14:31.827718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.827672    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podStartSLOduration=1.827658992 podStartE2EDuration="1.827658992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:31.821399211 +0000 UTC m=+1775.077578976" watchObservedRunningTime="2026-11-01 22:14:31.827658992 +0000 UTC m=+1775.083838754"
Nov 01 22:14:31.962377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:31.961497    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:32.965195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:32.965158    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:33.815851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:33.815812    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:33.816832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:33.816811    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:14:34.769135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.769084    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:14:34.784117 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784085    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784169    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784218    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784402    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784460    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784612    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784661    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784672    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:34.784810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784773    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.784895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.784878    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\") pod \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\" (UID: \"ee56283c-80a8-4cf3-a77a-0757a7bde40c\") "
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785043    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785188    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785340    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785758    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785782    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785797    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.786123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.785810    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.787513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.787477    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:14:34.787752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.787720    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:34.788503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.787882    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "kube-api-access-pkxqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:34.788906 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.788882    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:34.789424 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.789387    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket" pod "ee56283c-80a8-4cf3-a77a-0757a7bde40c" (UID: "ee56283c-80a8-4cf3-a77a-0757a7bde40c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:34.845597 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:34.845520    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:14:34.845750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.845727    5845 generic.go:356] "Generic (PLEG): container finished" podID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerID="41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7" exitCode=0
Nov 01 22:14:34.846436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.846415    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"
Nov 01 22:14:34.846729 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.846710    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" event={"ID":"ee56283c-80a8-4cf3-a77a-0757a7bde40c","Type":"ContainerDied","Data":"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"}
Nov 01 22:14:34.846801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.846756    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4" event={"ID":"ee56283c-80a8-4cf3-a77a-0757a7bde40c","Type":"ContainerDied","Data":"c2212a4f7ffa755634e33fd3119345010726794d3d4973551d24d60c47940dfc"}
Nov 01 22:14:34.846801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.846778    5845 scope.go:122] "RemoveContainer" containerID="41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"
Nov 01 22:14:34.862718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.862699    5845 scope.go:122] "RemoveContainer" containerID="41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"
Nov 01 22:14:34.863025 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:34.863005    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7\": container with ID starting with 41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7 not found: ID does not exist" containerID="41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"
Nov 01 22:14:34.863120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.863033    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7"} err="failed to get container status \"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7\": rpc error: code = NotFound desc = could not find container \"41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7\": container with ID starting with 41f2d5572feb3b9b4febdef00a17bb680bb575cb7acebf877dd4825431a888a7 not found: ID does not exist"
Nov 01 22:14:34.884771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.884746    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:14:34.888455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.888429    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.888633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.888458    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxqj\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-kube-api-access-pkxqj\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.888633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.888485    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.888633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.888498    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56283c-80a8-4cf3-a77a-0757a7bde40c-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.888633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.888511    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56283c-80a8-4cf3-a77a-0757a7bde40c-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:34.890655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:34.890634    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3577a0-openshift-default-84498fd47-bqmq4"]
Nov 01 22:14:36.864053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:36.864020    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" path="/var/lib/kubelet/pods/ee56283c-80a8-4cf3-a77a-0757a7bde40c/volumes"
Nov 01 22:14:41.708451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:41.708411    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:14:41.708864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:41.708411    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:14:42.689501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.689471    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:14:42.768789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.768767    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:14:42.781119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781093    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\" (UniqueName: \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\") pod \"dda8271f-8236-45c3-9c46-32f89a983da0\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") "
Nov 01 22:14:42.781236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781192    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\") pod \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") "
Nov 01 22:14:42.782093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781252    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\") pod \"dda8271f-8236-45c3-9c46-32f89a983da0\" (UID: \"dda8271f-8236-45c3-9c46-32f89a983da0\") "
Nov 01 22:14:42.782093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781402    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\" (UniqueName: \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\") pod \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\" (UID: \"50d44d0c-c985-4f79-8b90-5a3c7b5d190a\") "
Nov 01 22:14:42.782093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781581    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume" pod "50d44d0c-c985-4f79-8b90-5a3c7b5d190a" (UID: "50d44d0c-c985-4f79-8b90-5a3c7b5d190a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:42.782093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.781686    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume" pod "dda8271f-8236-45c3-9c46-32f89a983da0" (UID: "dda8271f-8236-45c3-9c46-32f89a983da0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:42.782393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.782139    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:42.782393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.782153    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda8271f-8236-45c3-9c46-32f89a983da0-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:42.783479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.783452    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx" pod "dda8271f-8236-45c3-9c46-32f89a983da0" (UID: "dda8271f-8236-45c3-9c46-32f89a983da0"). InnerVolumeSpecName "kube-api-access-zcghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:42.784206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.783560    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm" pod "50d44d0c-c985-4f79-8b90-5a3c7b5d190a" (UID: "50d44d0c-c985-4f79-8b90-5a3c7b5d190a"). InnerVolumeSpecName "kube-api-access-nfswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:42.894626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.894532    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfswm\" (UniqueName: \"kubernetes.io/projected/50d44d0c-c985-4f79-8b90-5a3c7b5d190a-kube-api-access-nfswm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:42.894626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.894575    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcghx\" (UniqueName: \"kubernetes.io/projected/dda8271f-8236-45c3-9c46-32f89a983da0-kube-api-access-zcghx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:42.976764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.976730    5845 generic.go:356] "Generic (PLEG): container finished" podID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerID="58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1" exitCode=0
Nov 01 22:14:42.976916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.976799    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"
Nov 01 22:14:42.976916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.976829    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" event={"ID":"50d44d0c-c985-4f79-8b90-5a3c7b5d190a","Type":"ContainerDied","Data":"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"}
Nov 01 22:14:42.976916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.976908    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx" event={"ID":"50d44d0c-c985-4f79-8b90-5a3c7b5d190a","Type":"ContainerDied","Data":"eef9db785d58f2f0cce02221fd276dc8bbb49a3e77605850b166e98d4f31d8f1"}
Nov 01 22:14:42.977066 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.976956    5845 scope.go:122] "RemoveContainer" containerID="58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"
Nov 01 22:14:42.978308 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.978270    5845 generic.go:356] "Generic (PLEG): container finished" podID="dda8271f-8236-45c3-9c46-32f89a983da0" containerID="904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618" exitCode=0
Nov 01 22:14:42.978437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.978365    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"
Nov 01 22:14:42.978493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.978360    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" event={"ID":"dda8271f-8236-45c3-9c46-32f89a983da0","Type":"ContainerDied","Data":"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"}
Nov 01 22:14:42.978493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.978475    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz" event={"ID":"dda8271f-8236-45c3-9c46-32f89a983da0","Type":"ContainerDied","Data":"358abad0f0b4748c4171c2bb5133eca5bed127327d9cefe41a64463c7d3f832f"}
Nov 01 22:14:42.990335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.990308    5845 scope.go:122] "RemoveContainer" containerID="58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"
Nov 01 22:14:42.990606 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:42.990585    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1\": container with ID starting with 58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1 not found: ID does not exist" containerID="58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"
Nov 01 22:14:42.990654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.990617    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1"} err="failed to get container status \"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1\": rpc error: code = NotFound desc = could not find container \"58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1\": container with ID starting with 58e2e1a1457ac5888b9f66c848f7a7807545d82ed6970b561bd6df84bd8f15b1 not found: ID does not exist"
Nov 01 22:14:42.990654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:42.990635    5845 scope.go:122] "RemoveContainer" containerID="904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"
Nov 01 22:14:43.004432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.004321    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:14:43.004493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.004467    5845 scope.go:122] "RemoveContainer" containerID="904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"
Nov 01 22:14:43.004750 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:43.004727    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618\": container with ID starting with 904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618 not found: ID does not exist" containerID="904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"
Nov 01 22:14:43.004814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.004759    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618"} err="failed to get container status \"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618\": rpc error: code = NotFound desc = could not find container \"904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618\": container with ID starting with 904a0257a782180080ad0728aacaaa8bd9569f95debc78bc402d755c9e296618 not found: ID does not exist"
Nov 01 22:14:43.012430 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.012401    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3577a0-f5df89dbc-fvvrx"]
Nov 01 22:14:43.024443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.024411    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:14:43.028578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:43.028557    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3577a0-d87647984-9xrbz"]
Nov 01 22:14:44.866258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:44.866223    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" path="/var/lib/kubelet/pods/50d44d0c-c985-4f79-8b90-5a3c7b5d190a/volumes"
Nov 01 22:14:44.866744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:44.866729    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" path="/var/lib/kubelet/pods/dda8271f-8236-45c3-9c46-32f89a983da0/volumes"
Nov 01 22:14:49.453452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:49.453421    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 22:14:49.453927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:49.453616    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy" containerID="cri-o://e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627" gracePeriod=30
Nov 01 22:14:49.597415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:49.596876    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 22:14:49.597415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:49.597134    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy" containerID="cri-o://0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9" gracePeriod=30
Nov 01 22:14:54.768127 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.768097    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 22:14:54.858337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858283    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858394    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858530    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858578    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858608    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858644    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.858996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.858878    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.859078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.859059    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.859162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.859131    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\") pod \"86223396-2925-4ad6-ac70-34ff71808484\" (UID: \"86223396-2925-4ad6-ac70-34ff71808484\") "
Nov 01 22:14:54.860247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.860221    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.860548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.860522    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.860752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.860728    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.860948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.860927    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.862199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.862163    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:54.866319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.866055    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:14:54.866425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.866358    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.871237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.871184    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:54.873376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.873350    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp" pod "86223396-2925-4ad6-ac70-34ff71808484" (UID: "86223396-2925-4ad6-ac70-34ff71808484"). InnerVolumeSpecName "kube-api-access-j8qxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:54.895938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.895914    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 22:14:54.960771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.960739    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.960945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.960827    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.960945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.960861    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.960958    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961100    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961217    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961278    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961482    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961539    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\") pod \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\" (UID: \"f2c3afe8-6835-4478-92fe-a8a0afddfed5\") "
Nov 01 22:14:54.961943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.961689    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.962072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962051    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962793    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962818    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86223396-2925-4ad6-ac70-34ff71808484-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962832    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962849    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962862    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962874    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8qxp\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-kube-api-access-j8qxp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.962893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962895    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86223396-2925-4ad6-ac70-34ff71808484-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962908    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86223396-2925-4ad6-ac70-34ff71808484-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962911    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962921    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962955    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962972    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.962990    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86223396-2925-4ad6-ac70-34ff71808484-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:54.963397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.963078    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:14:54.964788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.964765    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:14:54.964988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.964970    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "kube-api-access-hbrmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:54.965829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.965802    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:14:54.965933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:54.965845    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token" pod "f2c3afe8-6835-4478-92fe-a8a0afddfed5" (UID: "f2c3afe8-6835-4478-92fe-a8a0afddfed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:14:55.065781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065701    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.065781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065741    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.065781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065758    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.065781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065771    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.066022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065784    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbrmv\" (UniqueName: \"kubernetes.io/projected/f2c3afe8-6835-4478-92fe-a8a0afddfed5-kube-api-access-hbrmv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.066022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065800    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.066022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.065812    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c3afe8-6835-4478-92fe-a8a0afddfed5-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:14:55.096631 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.096598    5845 generic.go:356] "Generic (PLEG): container finished" podID="86223396-2925-4ad6-ac70-34ff71808484" containerID="e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627" exitCode=0
Nov 01 22:14:55.096780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.096682    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"
Nov 01 22:14:55.096780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.096689    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" event={"ID":"86223396-2925-4ad6-ac70-34ff71808484","Type":"ContainerDied","Data":"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"}
Nov 01 22:14:55.096899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.096784    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l" event={"ID":"86223396-2925-4ad6-ac70-34ff71808484","Type":"ContainerDied","Data":"f305c3759bc3650e0e7330aba5347769c74dd0f11cfb248fbb64cc0bf27f37ba"}
Nov 01 22:14:55.096899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.096805    5845 scope.go:122] "RemoveContainer" containerID="e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"
Nov 01 22:14:55.098651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.098623    5845 generic.go:356] "Generic (PLEG): container finished" podID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerID="0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9" exitCode=0
Nov 01 22:14:55.098788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.098700    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"
Nov 01 22:14:55.098880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.098707    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" event={"ID":"f2c3afe8-6835-4478-92fe-a8a0afddfed5","Type":"ContainerDied","Data":"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"}
Nov 01 22:14:55.098880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.098852    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl" event={"ID":"f2c3afe8-6835-4478-92fe-a8a0afddfed5","Type":"ContainerDied","Data":"afce0d20ff9adf1d6b8c7a186698397b9789b8a94f7356081f51b252637d029a"}
Nov 01 22:14:55.110233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.110211    5845 scope.go:122] "RemoveContainer" containerID="e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"
Nov 01 22:14:55.110641 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:55.110576    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627\": container with ID starting with e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627 not found: ID does not exist" containerID="e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"
Nov 01 22:14:55.110641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.110602    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627"} err="failed to get container status \"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627\": rpc error: code = NotFound desc = could not find container \"e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627\": container with ID starting with e2733a7aaa40baf36dfcc9db6a4c1e9c90cde240ebd05fd5a4c977b940c74627 not found: ID does not exist"
Nov 01 22:14:55.110641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.110618    5845 scope.go:122] "RemoveContainer" containerID="0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"
Nov 01 22:14:55.124239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.123465    5845 scope.go:122] "RemoveContainer" containerID="0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"
Nov 01 22:14:55.129115 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:14:55.128654    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9\": container with ID starting with 0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9 not found: ID does not exist" containerID="0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"
Nov 01 22:14:55.129115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.128791    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9"} err="failed to get container status \"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9\": rpc error: code = NotFound desc = could not find container \"0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9\": container with ID starting with 0ff2af2215576db26d8f2a20646445a9543c947cf380c7501622d2534aff8ae9 not found: ID does not exist"
Nov 01 22:14:55.133977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.133848    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 22:14:55.138949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.138931    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-lrtdl"]
Nov 01 22:14:55.248310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.238965    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 22:14:55.278327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:55.268274    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-dj75l"]
Nov 01 22:14:56.864012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:56.863982    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86223396-2925-4ad6-ac70-34ff71808484" path="/var/lib/kubelet/pods/86223396-2925-4ad6-ac70-34ff71808484/volumes"
Nov 01 22:14:56.864560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:56.864536    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" path="/var/lib/kubelet/pods/f2c3afe8-6835-4478-92fe-a8a0afddfed5/volumes"
Nov 01 22:14:57.185933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.185856    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:14:57.189153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.189130    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:14:57.308136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.308102    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:14:57.308965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.308948    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy"
Nov 01 22:14:57.309020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.308967    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy"
Nov 01 22:14:57.309020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.308979    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing"
Nov 01 22:14:57.309020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.308984    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing"
Nov 01 22:14:57.309020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309016    5845 cpu_manager.go:414] "containerMap: removing container" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy"
Nov 01 22:14:57.309143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309022    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy"
Nov 01 22:14:57.309143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309052    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy"
Nov 01 22:14:57.309143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309057    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy"
Nov 01 22:14:57.309143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309073    5845 cpu_manager.go:414] "containerMap: removing container" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing"
Nov 01 22:14:57.309143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309077    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing"
Nov 01 22:14:57.309285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309194    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="86223396-2925-4ad6-ac70-34ff71808484" containerName="istio-proxy"
Nov 01 22:14:57.309285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309207    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56283c-80a8-4cf3-a77a-0757a7bde40c" containerName="istio-proxy"
Nov 01 22:14:57.309285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309215    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d44d0c-c985-4f79-8b90-5a3c7b5d190a" containerName="payload-pre-processing"
Nov 01 22:14:57.309285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309222    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c3afe8-6835-4478-92fe-a8a0afddfed5" containerName="istio-proxy"
Nov 01 22:14:57.309285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.309228    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda8271f-8236-45c3-9c46-32f89a983da0" containerName="payload-processing"
Nov 01 22:14:57.314436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.314416    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.321573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.321526    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6b4c49\""
Nov 01 22:14:57.321573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.321564    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6b4c49-dockercfg-7rb7k\""
Nov 01 22:14:57.324528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.324501    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:14:57.367442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.367403    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnxws\" (UniqueName: \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.367614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.367474    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.372983 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.372947    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:14:57.381351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.380567    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:14:57.381351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.380789    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.492321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.489756    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.492321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.490002    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8sj\" (UniqueName: \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.492321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.490253    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnxws\" (UniqueName: \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.492321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.490336    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.515320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.506403    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.552321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.534231    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnxws\" (UniqueName: \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\") pod \"payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.605316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.599496    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8sj\" (UniqueName: \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.605316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.599930    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.609316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.608640    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.635325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.635104    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:57.639319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.637572    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8sj\" (UniqueName: \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\") pod \"payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:57.736001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:57.702669    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:58.048049 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:58.048017    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:14:58.050993 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:14:58.050962    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f063e31_bd21_40e7_a65e_ebe5e2fdafb2.slice/crio-b0e671f1b74301a8152813f9ef2a63a647b0fb449877c0c1c9493b624de343e6 WatchSource:0}: Error finding container b0e671f1b74301a8152813f9ef2a63a647b0fb449877c0c1c9493b624de343e6: Status 404 returned error can't find the container with id b0e671f1b74301a8152813f9ef2a63a647b0fb449877c0c1c9493b624de343e6
Nov 01 22:14:58.070050 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:14:58.070017    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2bdafc_355f_41b4_aa9c_11e32f374db3.slice/crio-67b4db864e9aa75fa8d485811913458241840b712166007b1289894d27d58be5 WatchSource:0}: Error finding container 67b4db864e9aa75fa8d485811913458241840b712166007b1289894d27d58be5: Status 404 returned error can't find the container with id 67b4db864e9aa75fa8d485811913458241840b712166007b1289894d27d58be5
Nov 01 22:14:58.071464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:58.071435    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:14:58.131102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:58.131068    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" event={"ID":"ee2bdafc-355f-41b4-aa9c-11e32f374db3","Type":"ContainerStarted","Data":"67b4db864e9aa75fa8d485811913458241840b712166007b1289894d27d58be5"}
Nov 01 22:14:58.132755 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:58.132731    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" event={"ID":"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2","Type":"ContainerStarted","Data":"b0e671f1b74301a8152813f9ef2a63a647b0fb449877c0c1c9493b624de343e6"}
Nov 01 22:14:59.143362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.143288    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" event={"ID":"ee2bdafc-355f-41b4-aa9c-11e32f374db3","Type":"ContainerStarted","Data":"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"}
Nov 01 22:14:59.143849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.143420    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:14:59.145350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.145280    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" event={"ID":"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2","Type":"ContainerStarted","Data":"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"}
Nov 01 22:14:59.145467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.145377    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:14:59.167600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.167555    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" podStartSLOduration=2.167542664 podStartE2EDuration="2.167542664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:59.162499039 +0000 UTC m=+1802.418678803" watchObservedRunningTime="2026-11-01 22:14:59.167542664 +0000 UTC m=+1802.423722428"
Nov 01 22:14:59.188258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:14:59.188216    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" podStartSLOduration=2.188199591 podStartE2EDuration="2.188199591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:14:59.182412213 +0000 UTC m=+1802.438591974" watchObservedRunningTime="2026-11-01 22:14:59.188199591 +0000 UTC m=+1802.444379357"
Nov 01 22:15:00.208236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.206664    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:00.214573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.214547    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.233556 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.233531    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:00.256367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.255139    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hshd\" (UniqueName: \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.256367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.255325    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.259598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.257487    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\""
Nov 01 22:15:00.276971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.275567    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:00.276971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.275619    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:00.276971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.275760    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.289046 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.285755    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:00.289046 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.285813    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:00.293151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.290821    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.299434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.299405    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.306078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.306049    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:00.317698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.317666    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:00.330122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.329677    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.361735    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.361820    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8z9\" (UniqueName: \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.362047    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.362192    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.362464    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqnm\" (UniqueName: \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.362581    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hshd\" (UniqueName: \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.362907    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflgw\" (UniqueName: \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.364381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.363167    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.364907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.364753    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.371448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.371421    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:00.372678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.372128    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.377976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.377948    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hshd\" (UniqueName: \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\") pod \"maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.385608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.385582    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:00.385761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.385626    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:00.385761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.385696    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.468996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.468968    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqnb\" (UniqueName: \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.469155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469036    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.469155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469097    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8z9\" (UniqueName: \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.469155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469127    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.469357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469202    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2sz\" (UniqueName: \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.469357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469286    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.469459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469396    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqnm\" (UniqueName: \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.469515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469484    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.469720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469694    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflgw\" (UniqueName: \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.469796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469756    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.469796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469788    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.469879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469861    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.470018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.469990    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.481874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.481855    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8z9\" (UniqueName: \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\") pod \"maas-api-key-cleanup-29892855-t8245\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.482281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.482262    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqnm\" (UniqueName: \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\") pod \"maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.482461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.482440    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflgw\" (UniqueName: \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\") pod \"maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.571926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.571891    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2sz\" (UniqueName: \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.572077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.572002    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.572167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.572150    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.572332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.572311    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqnb\" (UniqueName: \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.572752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.572731    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.572837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.572733    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.583051 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.583026    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:00.584061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.584042    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqnb\" (UniqueName: \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\") pod \"maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.587017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.586998    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2sz\" (UniqueName: \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.615998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.615970    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:00.655774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.655413    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:00.675942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.675708    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:00.716533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.716266    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:00.732599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.729193    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:00.924364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:00.924137    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:00.927043 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:15:00.927005    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a9e037_1694_474f_9acf_e71d4963930b.slice/crio-693acac8bf223906239cb550f578521052a48e94d0f440d176b6d31168dc5a82 WatchSource:0}: Error finding container 693acac8bf223906239cb550f578521052a48e94d0f440d176b6d31168dc5a82: Status 404 returned error can't find the container with id 693acac8bf223906239cb550f578521052a48e94d0f440d176b6d31168dc5a82
Nov 01 22:15:01.045371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.041206    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:01.075519 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.074936    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:01.095839 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.095772    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:01.096846 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:15:01.096819    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1def6a01_5d2b_4c1b_b3e9_4d7092975fce.slice/crio-f03362a69af30730a8d022f9194ed97375ea128cbbcd67a5f9cd20928c8c30fe WatchSource:0}: Error finding container f03362a69af30730a8d022f9194ed97375ea128cbbcd67a5f9cd20928c8c30fe: Status 404 returned error can't find the container with id f03362a69af30730a8d022f9194ed97375ea128cbbcd67a5f9cd20928c8c30fe
Nov 01 22:15:01.125112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.125089    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:01.130862 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:15:01.130838    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8f0ea5_3f87_4ced_a26a_eeed9a7e1446.slice/crio-424ed8aa3f51a5845fe5e91cdd73085481f12c6815d9348380f2d24ad15fccd4 WatchSource:0}: Error finding container 424ed8aa3f51a5845fe5e91cdd73085481f12c6815d9348380f2d24ad15fccd4: Status 404 returned error can't find the container with id 424ed8aa3f51a5845fe5e91cdd73085481f12c6815d9348380f2d24ad15fccd4
Nov 01 22:15:01.164851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.164822    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerStarted","Data":"f11ba7abc7de27cf6cd60fc33aa6577968116abf80641fedeed2a1fd0117981d"}
Nov 01 22:15:01.166306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.166270    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerStarted","Data":"8a22a6379f179b16a67433e4f8f6ba6cedf6353cf5b384a99aa79124a2a6f99f"}
Nov 01 22:15:01.167790 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.167765    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerStarted","Data":"f03362a69af30730a8d022f9194ed97375ea128cbbcd67a5f9cd20928c8c30fe"}
Nov 01 22:15:01.169187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.169163    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerStarted","Data":"424ed8aa3f51a5845fe5e91cdd73085481f12c6815d9348380f2d24ad15fccd4"}
Nov 01 22:15:01.170576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.170555    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerStarted","Data":"693acac8bf223906239cb550f578521052a48e94d0f440d176b6d31168dc5a82"}
Nov 01 22:15:01.187360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:01.187337    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:01.251350 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:15:01.251230    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33be8278_82a2_49cd_abc9_43baf840e5ea.slice/crio-8068060b5efb0ba4a36e6ad313736210bd3f5248358191b6b202e406dd7a37e5 WatchSource:0}: Error finding container 8068060b5efb0ba4a36e6ad313736210bd3f5248358191b6b202e406dd7a37e5: Status 404 returned error can't find the container with id 8068060b5efb0ba4a36e6ad313736210bd3f5248358191b6b202e406dd7a37e5
Nov 01 22:15:02.181879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.181788    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerStarted","Data":"e733c725413e3470e762729fc15a9e072247c200c2ab0c646b1da7c116a4773b"}
Nov 01 22:15:02.183488 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.183461    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerStarted","Data":"550b1113f3fcfbf7bb8932a06db5837a51e8234d68c5ff80276ba2e0b7c1d9c9"}
Nov 01 22:15:02.185044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.185022    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerStarted","Data":"b7b413838e42189370655a3d75c4104a339c035216274e6d80224af464754b84"}
Nov 01 22:15:02.185133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.185057    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerStarted","Data":"8068060b5efb0ba4a36e6ad313736210bd3f5248358191b6b202e406dd7a37e5"}
Nov 01 22:15:02.186658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.186633    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerStarted","Data":"41cacc32acf9fc561919bf93123f90cba3ce84ffac971fc71f0030383c5e70a4"}
Nov 01 22:15:02.188450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.188430    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerStarted","Data":"ff6e68449fcac5cafa4bc84139f48fa8b94f4aa5ac5003673bb22c4ed02a7709"}
Nov 01 22:15:02.190015 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.189989    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerStarted","Data":"82e5851273681494417f1e789624bedb5a1bed7dfa7a6b8aec7cb4dd94deb99c"}
Nov 01 22:15:02.204778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.204729    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" podStartSLOduration=2.204717178 podStartE2EDuration="2.204717178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.201000166 +0000 UTC m=+1805.457179932" watchObservedRunningTime="2026-11-01 22:15:02.204717178 +0000 UTC m=+1805.460896940"
Nov 01 22:15:02.223019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.222972    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" podStartSLOduration=2.22296132 podStartE2EDuration="2.22296132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.221395419 +0000 UTC m=+1805.477575179" watchObservedRunningTime="2026-11-01 22:15:02.22296132 +0000 UTC m=+1805.479141101"
Nov 01 22:15:02.266954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.266896    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" podStartSLOduration=2.26687517 podStartE2EDuration="2.26687517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.248722448 +0000 UTC m=+1805.504902212" watchObservedRunningTime="2026-11-01 22:15:02.26687517 +0000 UTC m=+1805.523054931"
Nov 01 22:15:02.301056 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.301005    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" podStartSLOduration=2.300991197 podStartE2EDuration="2.300991197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.290874681 +0000 UTC m=+1805.547054447" watchObservedRunningTime="2026-11-01 22:15:02.300991197 +0000 UTC m=+1805.557170960"
Nov 01 22:15:02.316925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.316877    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" podStartSLOduration=2.316863069 podStartE2EDuration="2.316863069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.313584514 +0000 UTC m=+1805.569764289" watchObservedRunningTime="2026-11-01 22:15:02.316863069 +0000 UTC m=+1805.573042831"
Nov 01 22:15:02.337958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:02.337900    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" podStartSLOduration=2.337885186 podStartE2EDuration="2.337885186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:15:02.332500331 +0000 UTC m=+1805.588680095" watchObservedRunningTime="2026-11-01 22:15:02.337885186 +0000 UTC m=+1805.594064948"
Nov 01 22:15:03.200238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.200199    5845 generic.go:356] "Generic (PLEG): container finished" podID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerID="b7b413838e42189370655a3d75c4104a339c035216274e6d80224af464754b84" exitCode=60
Nov 01 22:15:03.200454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.200291    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerDied","Data":"b7b413838e42189370655a3d75c4104a339c035216274e6d80224af464754b84"}
Nov 01 22:15:03.200899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.200861    5845 scope.go:122] "RemoveContainer" containerID="b7b413838e42189370655a3d75c4104a339c035216274e6d80224af464754b84"
Nov 01 22:15:03.202992 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.202965    5845 generic.go:356] "Generic (PLEG): container finished" podID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerID="41cacc32acf9fc561919bf93123f90cba3ce84ffac971fc71f0030383c5e70a4" exitCode=60
Nov 01 22:15:03.203123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.203044    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerDied","Data":"41cacc32acf9fc561919bf93123f90cba3ce84ffac971fc71f0030383c5e70a4"}
Nov 01 22:15:03.203344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.203328    5845 scope.go:122] "RemoveContainer" containerID="41cacc32acf9fc561919bf93123f90cba3ce84ffac971fc71f0030383c5e70a4"
Nov 01 22:15:03.206094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.206071    5845 generic.go:356] "Generic (PLEG): container finished" podID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerID="ff6e68449fcac5cafa4bc84139f48fa8b94f4aa5ac5003673bb22c4ed02a7709" exitCode=60
Nov 01 22:15:03.206198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.206183    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerDied","Data":"ff6e68449fcac5cafa4bc84139f48fa8b94f4aa5ac5003673bb22c4ed02a7709"}
Nov 01 22:15:03.206539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.206519    5845 scope.go:122] "RemoveContainer" containerID="ff6e68449fcac5cafa4bc84139f48fa8b94f4aa5ac5003673bb22c4ed02a7709"
Nov 01 22:15:03.208193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.208168    5845 generic.go:356] "Generic (PLEG): container finished" podID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerID="82e5851273681494417f1e789624bedb5a1bed7dfa7a6b8aec7cb4dd94deb99c" exitCode=60
Nov 01 22:15:03.208373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.208248    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerDied","Data":"82e5851273681494417f1e789624bedb5a1bed7dfa7a6b8aec7cb4dd94deb99c"}
Nov 01 22:15:03.208611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.208593    5845 scope.go:122] "RemoveContainer" containerID="82e5851273681494417f1e789624bedb5a1bed7dfa7a6b8aec7cb4dd94deb99c"
Nov 01 22:15:03.210338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.210288    5845 generic.go:356] "Generic (PLEG): container finished" podID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerID="e733c725413e3470e762729fc15a9e072247c200c2ab0c646b1da7c116a4773b" exitCode=60
Nov 01 22:15:03.210444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.210383    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerDied","Data":"e733c725413e3470e762729fc15a9e072247c200c2ab0c646b1da7c116a4773b"}
Nov 01 22:15:03.210629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.210614    5845 scope.go:122] "RemoveContainer" containerID="e733c725413e3470e762729fc15a9e072247c200c2ab0c646b1da7c116a4773b"
Nov 01 22:15:03.212081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.212051    5845 generic.go:356] "Generic (PLEG): container finished" podID="73a9e037-1694-474f-9acf-e71d4963930b" containerID="550b1113f3fcfbf7bb8932a06db5837a51e8234d68c5ff80276ba2e0b7c1d9c9" exitCode=60
Nov 01 22:15:03.212159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.212105    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerDied","Data":"550b1113f3fcfbf7bb8932a06db5837a51e8234d68c5ff80276ba2e0b7c1d9c9"}
Nov 01 22:15:03.212289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:03.212276    5845 scope.go:122] "RemoveContainer" containerID="550b1113f3fcfbf7bb8932a06db5837a51e8234d68c5ff80276ba2e0b7c1d9c9"
Nov 01 22:15:04.223555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.223520    5845 generic.go:356] "Generic (PLEG): container finished" podID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerID="49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528" exitCode=60
Nov 01 22:15:04.223967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.223602    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerDied","Data":"49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528"}
Nov 01 22:15:04.223967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.223669    5845 scope.go:122] "RemoveContainer" containerID="b7b413838e42189370655a3d75c4104a339c035216274e6d80224af464754b84"
Nov 01 22:15:04.224096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.223974    5845 scope.go:122] "RemoveContainer" containerID="49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528"
Nov 01 22:15:04.224448 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:04.224423    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv_odh-ai-gateway-infra(33be8278-82a2-49cd-abc9-43baf840e5ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea"
Nov 01 22:15:04.225878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.225854    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerStarted","Data":"c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"}
Nov 01 22:15:04.227831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.227805    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerStarted","Data":"a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"}
Nov 01 22:15:04.229920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.229898    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerStarted","Data":"e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"}
Nov 01 22:15:04.232651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.232619    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerStarted","Data":"a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"}
Nov 01 22:15:04.234529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:04.234507    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerStarted","Data":"016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"}
Nov 01 22:15:04.817714 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:04.817669    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:15:05.259164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.259123    5845 generic.go:356] "Generic (PLEG): container finished" podID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerID="a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67" exitCode=60
Nov 01 22:15:05.259597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.259237    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerDied","Data":"a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"}
Nov 01 22:15:05.259597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.259310    5845 scope.go:122] "RemoveContainer" containerID="e733c725413e3470e762729fc15a9e072247c200c2ab0c646b1da7c116a4773b"
Nov 01 22:15:05.259778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.259761    5845 scope.go:122] "RemoveContainer" containerID="a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"
Nov 01 22:15:05.260164 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.260142    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w_odh-ai-gateway-infra(af8f0ea5-3f87-4ced-a26a-eeed9a7e1446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446"
Nov 01 22:15:05.263486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.263461    5845 generic.go:356] "Generic (PLEG): container finished" podID="73a9e037-1694-474f-9acf-e71d4963930b" containerID="016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c" exitCode=60
Nov 01 22:15:05.263590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.263557    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerDied","Data":"016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"}
Nov 01 22:15:05.263935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.263914    5845 scope.go:122] "RemoveContainer" containerID="016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"
Nov 01 22:15:05.264456 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.264429    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp_odh-ai-gateway-infra(73a9e037-1694-474f-9acf-e71d4963930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" podUID="73a9e037-1694-474f-9acf-e71d4963930b"
Nov 01 22:15:05.266239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.266216    5845 scope.go:122] "RemoveContainer" containerID="49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528"
Nov 01 22:15:05.266633 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.266611    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv_odh-ai-gateway-infra(33be8278-82a2-49cd-abc9-43baf840e5ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea"
Nov 01 22:15:05.268999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.268251    5845 generic.go:356] "Generic (PLEG): container finished" podID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerID="c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7" exitCode=60
Nov 01 22:15:05.269104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.269086    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerDied","Data":"c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"}
Nov 01 22:15:05.269486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.269456    5845 scope.go:122] "RemoveContainer" containerID="c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"
Nov 01 22:15:05.269831 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.269808    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7_odh-ai-gateway-infra(61eb1c81-a220-4b85-80a0-1e15e6a57b02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02"
Nov 01 22:15:05.272312 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.272273    5845 generic.go:356] "Generic (PLEG): container finished" podID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerID="a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385" exitCode=60
Nov 01 22:15:05.272396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.272325    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerDied","Data":"a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"}
Nov 01 22:15:05.272695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.272677    5845 scope.go:122] "RemoveContainer" containerID="a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"
Nov 01 22:15:05.273047 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.273025    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892855-t8245_odh-ai-gateway-infra(1f9b1317-41d8-46d9-8989-ff01da63c54f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f"
Nov 01 22:15:05.274596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.274484    5845 generic.go:356] "Generic (PLEG): container finished" podID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerID="e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7" exitCode=60
Nov 01 22:15:05.274596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.274513    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerDied","Data":"e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"}
Nov 01 22:15:05.274798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.274783    5845 scope.go:122] "RemoveContainer" containerID="e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"
Nov 01 22:15:05.275100 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:05.275081    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89_odh-ai-gateway-infra(1def6a01-5d2b-4c1b-b3e9-4d7092975fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce"
Nov 01 22:15:05.277712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.277692    5845 scope.go:122] "RemoveContainer" containerID="550b1113f3fcfbf7bb8932a06db5837a51e8234d68c5ff80276ba2e0b7c1d9c9"
Nov 01 22:15:05.293326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.293257    5845 scope.go:122] "RemoveContainer" containerID="41cacc32acf9fc561919bf93123f90cba3ce84ffac971fc71f0030383c5e70a4"
Nov 01 22:15:05.308853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.308357    5845 scope.go:122] "RemoveContainer" containerID="ff6e68449fcac5cafa4bc84139f48fa8b94f4aa5ac5003673bb22c4ed02a7709"
Nov 01 22:15:05.343540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:05.342216    5845 scope.go:122] "RemoveContainer" containerID="82e5851273681494417f1e789624bedb5a1bed7dfa7a6b8aec7cb4dd94deb99c"
Nov 01 22:15:06.296078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:06.296041    5845 scope.go:122] "RemoveContainer" containerID="c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"
Nov 01 22:15:06.296679 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:06.296578    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7_odh-ai-gateway-infra(61eb1c81-a220-4b85-80a0-1e15e6a57b02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02"
Nov 01 22:15:06.299371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:06.299345    5845 scope.go:122] "RemoveContainer" containerID="a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"
Nov 01 22:15:06.299753 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:06.299725    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892855-t8245_odh-ai-gateway-infra(1f9b1317-41d8-46d9-8989-ff01da63c54f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f"
Nov 01 22:15:06.301999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:06.301979    5845 scope.go:122] "RemoveContainer" containerID="e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"
Nov 01 22:15:06.302342 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:06.302277    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89_odh-ai-gateway-infra(1def6a01-5d2b-4c1b-b3e9-4d7092975fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce"
Nov 01 22:15:06.303794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:06.303776    5845 scope.go:122] "RemoveContainer" containerID="a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"
Nov 01 22:15:06.304054 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:06.304034    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w_odh-ai-gateway-infra(af8f0ea5-3f87-4ced-a26a-eeed9a7e1446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446"
Nov 01 22:15:06.305372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:06.305350    5845 scope.go:122] "RemoveContainer" containerID="016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"
Nov 01 22:15:06.305651 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:06.305630    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp_odh-ai-gateway-infra(73a9e037-1694-474f-9acf-e71d4963930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" podUID="73a9e037-1694-474f-9acf-e71d4963930b"
Nov 01 22:15:10.157002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:10.156916    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:15:10.157493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:10.157323    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:15:15.854447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:15.854416    5845 scope.go:122] "RemoveContainer" containerID="c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"
Nov 01 22:15:16.443106 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:16.443069    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerStarted","Data":"af7c7067e519a540517a6b500fa0cd157c2b7a190ebc10bf5f6f007a560476f3"}
Nov 01 22:15:16.861911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:16.861885    5845 scope.go:122] "RemoveContainer" containerID="e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"
Nov 01 22:15:17.483568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.482134    5845 generic.go:356] "Generic (PLEG): container finished" podID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerID="af7c7067e519a540517a6b500fa0cd157c2b7a190ebc10bf5f6f007a560476f3" exitCode=60
Nov 01 22:15:17.483568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.482343    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerDied","Data":"af7c7067e519a540517a6b500fa0cd157c2b7a190ebc10bf5f6f007a560476f3"}
Nov 01 22:15:17.483568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.482435    5845 scope.go:122] "RemoveContainer" containerID="c21fc440a563d50c4ec6c06ef54f2bd026646c9931c2aaab111aaac0a5a02ac7"
Nov 01 22:15:17.483568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.483069    5845 scope.go:122] "RemoveContainer" containerID="af7c7067e519a540517a6b500fa0cd157c2b7a190ebc10bf5f6f007a560476f3"
Nov 01 22:15:17.483568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.483439    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:17.484332 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:17.484039    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7_odh-ai-gateway-infra(61eb1c81-a220-4b85-80a0-1e15e6a57b02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02"
Nov 01 22:15:17.489274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.489236    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerStarted","Data":"82384ead50d701ad8202bb6264ecf48b9670e77ec37e0530089fc2ac2e335930"}
Nov 01 22:15:17.854091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.854065    5845 scope.go:122] "RemoveContainer" containerID="a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"
Nov 01 22:15:17.854271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:17.854251    5845 scope.go:122] "RemoveContainer" containerID="016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"
Nov 01 22:15:18.513394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.513347    5845 generic.go:356] "Generic (PLEG): container finished" podID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerID="82384ead50d701ad8202bb6264ecf48b9670e77ec37e0530089fc2ac2e335930" exitCode=60
Nov 01 22:15:18.514157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.513526    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerDied","Data":"82384ead50d701ad8202bb6264ecf48b9670e77ec37e0530089fc2ac2e335930"}
Nov 01 22:15:18.514157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.513584    5845 scope.go:122] "RemoveContainer" containerID="e48d11b89e6ee895984f54b6f3841d839c1c1f77ddd0721a0b18bc0cb2afbef7"
Nov 01 22:15:18.514157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.514118    5845 scope.go:122] "RemoveContainer" containerID="82384ead50d701ad8202bb6264ecf48b9670e77ec37e0530089fc2ac2e335930"
Nov 01 22:15:18.514699 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:18.514612    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89_odh-ai-gateway-infra(1def6a01-5d2b-4c1b-b3e9-4d7092975fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce"
Nov 01 22:15:18.521146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.521060    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerStarted","Data":"3146a41ee04b94c32a5784b350507535789096c0dc7ee228e19d12bac6cb5d52"}
Nov 01 22:15:18.529714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.529690    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerStarted","Data":"56ec9275e19de9f6c8614f867a2e954fec23c755c32a71ab028f697497245dc0"}
Nov 01 22:15:18.554563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.554532    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:18.793854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.793826    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:18.825460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.825431    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\" (UniqueName: \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\") pod \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") "
Nov 01 22:15:18.825740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.825723    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\") pod \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\" (UID: \"61eb1c81-a220-4b85-80a0-1e15e6a57b02\") "
Nov 01 22:15:18.826133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.826109    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca" pod "61eb1c81-a220-4b85-80a0-1e15e6a57b02" (UID: "61eb1c81-a220-4b85-80a0-1e15e6a57b02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:18.826580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.826557    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61eb1c81-a220-4b85-80a0-1e15e6a57b02-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:18.828135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.828107    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm" pod "61eb1c81-a220-4b85-80a0-1e15e6a57b02" (UID: "61eb1c81-a220-4b85-80a0-1e15e6a57b02"). InnerVolumeSpecName "kube-api-access-nvqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:18.888321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.881844    5845 scope.go:122] "RemoveContainer" containerID="a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"
Nov 01 22:15:18.914320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.896987    5845 scope.go:122] "RemoveContainer" containerID="49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528"
Nov 01 22:15:18.943556 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:18.941104    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvqnm\" (UniqueName: \"kubernetes.io/projected/61eb1c81-a220-4b85-80a0-1e15e6a57b02-kube-api-access-nvqnm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:19.540496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.540465    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerStarted","Data":"ff47438ff51521da9bf002da2465df12db7d11350730ee84532db2d05813d42e"}
Nov 01 22:15:19.541959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.541936    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"
Nov 01 22:15:19.542088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.541959    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7" event={"ID":"61eb1c81-a220-4b85-80a0-1e15e6a57b02","Type":"ContainerDied","Data":"f11ba7abc7de27cf6cd60fc33aa6577968116abf80641fedeed2a1fd0117981d"}
Nov 01 22:15:19.542088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.542013    5845 scope.go:122] "RemoveContainer" containerID="af7c7067e519a540517a6b500fa0cd157c2b7a190ebc10bf5f6f007a560476f3"
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.544280    5845 generic.go:356] "Generic (PLEG): container finished" podID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerID="56ec9275e19de9f6c8614f867a2e954fec23c755c32a71ab028f697497245dc0" exitCode=60
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.544338    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerDied","Data":"56ec9275e19de9f6c8614f867a2e954fec23c755c32a71ab028f697497245dc0"}
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.545915    5845 scope.go:122] "RemoveContainer" containerID="56ec9275e19de9f6c8614f867a2e954fec23c755c32a71ab028f697497245dc0"
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:19.546276    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892855-t8245_odh-ai-gateway-infra(1f9b1317-41d8-46d9-8989-ff01da63c54f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f"
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.550495    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerStarted","Data":"fa658224e02e1cd790afc7d4ed35c5b90c8c90bdc10e92cc328047b384e72046"}
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.552672    5845 generic.go:356] "Generic (PLEG): container finished" podID="73a9e037-1694-474f-9acf-e71d4963930b" containerID="3146a41ee04b94c32a5784b350507535789096c0dc7ee228e19d12bac6cb5d52" exitCode=60
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.552729    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerDied","Data":"3146a41ee04b94c32a5784b350507535789096c0dc7ee228e19d12bac6cb5d52"}
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.553017    5845 scope.go:122] "RemoveContainer" containerID="3146a41ee04b94c32a5784b350507535789096c0dc7ee228e19d12bac6cb5d52"
Nov 01 22:15:19.557362 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:19.553224    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp_odh-ai-gateway-infra(73a9e037-1694-474f-9acf-e71d4963930b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" podUID="73a9e037-1694-474f-9acf-e71d4963930b"
Nov 01 22:15:19.558581 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.558564    5845 scope.go:122] "RemoveContainer" containerID="a29e7a5227fed6103c1d78dfbadc30fdb2fdca5ab44eec9502239992c0e50385"
Nov 01 22:15:19.580415 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.579644    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:19.584628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.584607    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:19.585050 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.585025    5845 scope.go:122] "RemoveContainer" containerID="016c73c8dfc105b1032412850e5c599b0ef9b0aa5c704fd61e8d8481cde4e06c"
Nov 01 22:15:19.607362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.603136    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ff3e70-29892855-kdzv7"]
Nov 01 22:15:19.635745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.635714    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:19.771680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.771647    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:19.856722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.856620    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\") pod \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") "
Nov 01 22:15:19.856872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.856743    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\" (UniqueName: \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\") pod \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\" (UID: \"1def6a01-5d2b-4c1b-b3e9-4d7092975fce\") "
Nov 01 22:15:19.856997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.856973    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca" pod "1def6a01-5d2b-4c1b-b3e9-4d7092975fce" (UID: "1def6a01-5d2b-4c1b-b3e9-4d7092975fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:19.857260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.857237    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:19.858982 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.858957    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb" pod "1def6a01-5d2b-4c1b-b3e9-4d7092975fce" (UID: "1def6a01-5d2b-4c1b-b3e9-4d7092975fce"). InnerVolumeSpecName "kube-api-access-cvqnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:19.960598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:19.960565    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqnb\" (UniqueName: \"kubernetes.io/projected/1def6a01-5d2b-4c1b-b3e9-4d7092975fce-kube-api-access-cvqnb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:20.563224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.562981    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"
Nov 01 22:15:20.563224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.563149    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89" event={"ID":"1def6a01-5d2b-4c1b-b3e9-4d7092975fce","Type":"ContainerDied","Data":"f03362a69af30730a8d022f9194ed97375ea128cbbcd67a5f9cd20928c8c30fe"}
Nov 01 22:15:20.563915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.563232    5845 scope.go:122] "RemoveContainer" containerID="82384ead50d701ad8202bb6264ecf48b9670e77ec37e0530089fc2ac2e335930"
Nov 01 22:15:20.569150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.569126    5845 generic.go:356] "Generic (PLEG): container finished" podID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerID="fa658224e02e1cd790afc7d4ed35c5b90c8c90bdc10e92cc328047b384e72046" exitCode=60
Nov 01 22:15:20.569280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.569198    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerDied","Data":"fa658224e02e1cd790afc7d4ed35c5b90c8c90bdc10e92cc328047b384e72046"}
Nov 01 22:15:20.569708 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.569686    5845 scope.go:122] "RemoveContainer" containerID="fa658224e02e1cd790afc7d4ed35c5b90c8c90bdc10e92cc328047b384e72046"
Nov 01 22:15:20.570078 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:20.570052    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w_odh-ai-gateway-infra(af8f0ea5-3f87-4ced-a26a-eeed9a7e1446)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446"
Nov 01 22:15:20.574943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.574922    5845 generic.go:356] "Generic (PLEG): container finished" podID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerID="ff47438ff51521da9bf002da2465df12db7d11350730ee84532db2d05813d42e" exitCode=60
Nov 01 22:15:20.575048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.574996    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerDied","Data":"ff47438ff51521da9bf002da2465df12db7d11350730ee84532db2d05813d42e"}
Nov 01 22:15:20.575408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.575389    5845 scope.go:122] "RemoveContainer" containerID="ff47438ff51521da9bf002da2465df12db7d11350730ee84532db2d05813d42e"
Nov 01 22:15:20.575712 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:20.575688    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv_odh-ai-gateway-infra(33be8278-82a2-49cd-abc9-43baf840e5ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea"
Nov 01 22:15:20.597942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.579033    5845 scope.go:122] "RemoveContainer" containerID="a7c70f0d2ad5f4c460ec0a354195fef0be8d3571cddc89270048865ad15b3d67"
Nov 01 22:15:20.597942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.585004    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:20.669546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.669510    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:20.681652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.681616    5845 scope.go:122] "RemoveContainer" containerID="49a6dc81092a846ec162cdb3aa39d07c7c618abae5f3a96b538bc1af8424c528"
Nov 01 22:15:20.687469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.687441    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:20.695022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:20.694997    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e7e2df-29892855-s6w89"]
Nov 01 22:15:21.046242 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.046212    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" path="/var/lib/kubelet/pods/1def6a01-5d2b-4c1b-b3e9-4d7092975fce/volumes"
Nov 01 22:15:21.046956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.046928    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" path="/var/lib/kubelet/pods/61eb1c81-a220-4b85-80a0-1e15e6a57b02/volumes"
Nov 01 22:15:21.125546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.125518    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:21.212340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.209618    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\") pod \"73a9e037-1694-474f-9acf-e71d4963930b\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") "
Nov 01 22:15:21.212340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.210107    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\" (UniqueName: \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\") pod \"73a9e037-1694-474f-9acf-e71d4963930b\" (UID: \"73a9e037-1694-474f-9acf-e71d4963930b\") "
Nov 01 22:15:21.221032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.220421    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd" pod "73a9e037-1694-474f-9acf-e71d4963930b" (UID: "73a9e037-1694-474f-9acf-e71d4963930b"). InnerVolumeSpecName "kube-api-access-5hshd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:21.228748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.228708    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca" pod "73a9e037-1694-474f-9acf-e71d4963930b" (UID: "73a9e037-1694-474f-9acf-e71d4963930b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:21.231757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.231729    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:21.311560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.311474    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\" (UniqueName: \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\") pod \"1f9b1317-41d8-46d9-8989-ff01da63c54f\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") "
Nov 01 22:15:21.311850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.311822    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\") pod \"1f9b1317-41d8-46d9-8989-ff01da63c54f\" (UID: \"1f9b1317-41d8-46d9-8989-ff01da63c54f\") "
Nov 01 22:15:21.312457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.312287    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca" pod "1f9b1317-41d8-46d9-8989-ff01da63c54f" (UID: "1f9b1317-41d8-46d9-8989-ff01da63c54f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:21.312686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.312665    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9b1317-41d8-46d9-8989-ff01da63c54f-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:21.312762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.312692    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hshd\" (UniqueName: \"kubernetes.io/projected/73a9e037-1694-474f-9acf-e71d4963930b-kube-api-access-5hshd\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:21.312762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.312709    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a9e037-1694-474f-9acf-e71d4963930b-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:21.315952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.315925    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9" pod "1f9b1317-41d8-46d9-8989-ff01da63c54f" (UID: "1f9b1317-41d8-46d9-8989-ff01da63c54f"). InnerVolumeSpecName "kube-api-access-lv8z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:21.413869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.413829    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8z9\" (UniqueName: \"kubernetes.io/projected/1f9b1317-41d8-46d9-8989-ff01da63c54f-kube-api-access-lv8z9\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:21.658395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.658114    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245" event={"ID":"1f9b1317-41d8-46d9-8989-ff01da63c54f","Type":"ContainerDied","Data":"8a22a6379f179b16a67433e4f8f6ba6cedf6353cf5b384a99aa79124a2a6f99f"}
Nov 01 22:15:21.658395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.658229    5845 scope.go:122] "RemoveContainer" containerID="56ec9275e19de9f6c8614f867a2e954fec23c755c32a71ab028f697497245dc0"
Nov 01 22:15:21.659881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.658996    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"
Nov 01 22:15:21.668414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.667920    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp" event={"ID":"73a9e037-1694-474f-9acf-e71d4963930b","Type":"ContainerDied","Data":"693acac8bf223906239cb550f578521052a48e94d0f440d176b6d31168dc5a82"}
Nov 01 22:15:21.668650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.668359    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"
Nov 01 22:15:21.679498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.678508    5845 scope.go:122] "RemoveContainer" containerID="3146a41ee04b94c32a5784b350507535789096c0dc7ee228e19d12bac6cb5d52"
Nov 01 22:15:21.824604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.796617    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:21.824604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.811361    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892855-t8245"]
Nov 01 22:15:21.845317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.841921    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:21.845317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.842001    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9d8b5d-29892855-rxztp"]
Nov 01 22:15:21.974204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:21.973373    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:22.024621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.024590    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:22.106664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.105533    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\") pod \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") "
Nov 01 22:15:22.106664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.105680    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\") pod \"33be8278-82a2-49cd-abc9-43baf840e5ea\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") "
Nov 01 22:15:22.106664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.105729    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\" (UniqueName: \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\") pod \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\" (UID: \"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446\") "
Nov 01 22:15:22.107859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.107818    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca" pod "af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" (UID: "af8f0ea5-3f87-4ced-a26a-eeed9a7e1446"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:22.112759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.108248    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca" pod "33be8278-82a2-49cd-abc9-43baf840e5ea" (UID: "33be8278-82a2-49cd-abc9-43baf840e5ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:22.114509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.114437    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\" (UniqueName: \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\") pod \"33be8278-82a2-49cd-abc9-43baf840e5ea\" (UID: \"33be8278-82a2-49cd-abc9-43baf840e5ea\") "
Nov 01 22:15:22.127746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.120707    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz" pod "af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" (UID: "af8f0ea5-3f87-4ced-a26a-eeed9a7e1446"). InnerVolumeSpecName "kube-api-access-fv2sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:22.127746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.120778    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:22.127746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.120801    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33be8278-82a2-49cd-abc9-43baf840e5ea-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:22.159146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.142062    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw" pod "33be8278-82a2-49cd-abc9-43baf840e5ea" (UID: "33be8278-82a2-49cd-abc9-43baf840e5ea"). InnerVolumeSpecName "kube-api-access-rflgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:22.223012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.222975    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv2sz\" (UniqueName: \"kubernetes.io/projected/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446-kube-api-access-fv2sz\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:22.223199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.223016    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflgw\" (UniqueName: \"kubernetes.io/projected/33be8278-82a2-49cd-abc9-43baf840e5ea-kube-api-access-rflgw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:22.683670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.683581    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"
Nov 01 22:15:22.683670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.683612    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv" event={"ID":"33be8278-82a2-49cd-abc9-43baf840e5ea","Type":"ContainerDied","Data":"8068060b5efb0ba4a36e6ad313736210bd3f5248358191b6b202e406dd7a37e5"}
Nov 01 22:15:22.684162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.683687    5845 scope.go:122] "RemoveContainer" containerID="ff47438ff51521da9bf002da2465df12db7d11350730ee84532db2d05813d42e"
Nov 01 22:15:22.687343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.687318    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"
Nov 01 22:15:22.687527 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.687358    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w" event={"ID":"af8f0ea5-3f87-4ced-a26a-eeed9a7e1446","Type":"ContainerDied","Data":"424ed8aa3f51a5845fe5e91cdd73085481f12c6815d9348380f2d24ad15fccd4"}
Nov 01 22:15:22.698870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.698853    5845 scope.go:122] "RemoveContainer" containerID="fa658224e02e1cd790afc7d4ed35c5b90c8c90bdc10e92cc328047b384e72046"
Nov 01 22:15:22.719091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.719069    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:22.723883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.723861    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f033b1-29892855-htwlv"]
Nov 01 22:15:22.734540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.734518    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:22.743604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.743584    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-702002-29892855-tpz7w"]
Nov 01 22:15:22.918094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.918058    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" path="/var/lib/kubelet/pods/1f9b1317-41d8-46d9-8989-ff01da63c54f/volumes"
Nov 01 22:15:22.918858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.918832    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" path="/var/lib/kubelet/pods/33be8278-82a2-49cd-abc9-43baf840e5ea/volumes"
Nov 01 22:15:22.919600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.919573    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a9e037-1694-474f-9acf-e71d4963930b" path="/var/lib/kubelet/pods/73a9e037-1694-474f-9acf-e71d4963930b/volumes"
Nov 01 22:15:22.920675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:22.920651    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" path="/var/lib/kubelet/pods/af8f0ea5-3f87-4ced-a26a-eeed9a7e1446/volumes"
Nov 01 22:15:34.819605 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:34.819566    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:15:43.910991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:43.910947    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:15:44.606115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:44.606084    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:15:44.623170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:44.623127    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:15:48.911976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:48.911935    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerName="payload-processing" containerID="cri-o://c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc" gracePeriod=25
Nov 01 22:15:49.286969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.286944    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:15:49.402198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.402164    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\") pod \"3a5fdb64-a54c-4184-900a-136a92af01ed\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") "
Nov 01 22:15:49.402861 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.402243    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\" (UniqueName: \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\") pod \"3a5fdb64-a54c-4184-900a-136a92af01ed\" (UID: \"3a5fdb64-a54c-4184-900a-136a92af01ed\") "
Nov 01 22:15:49.402861 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.402616    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume" pod "3a5fdb64-a54c-4184-900a-136a92af01ed" (UID: "3a5fdb64-a54c-4184-900a-136a92af01ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:49.403099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.403079    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5fdb64-a54c-4184-900a-136a92af01ed-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:49.411859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.411819    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b" pod "3a5fdb64-a54c-4184-900a-136a92af01ed" (UID: "3a5fdb64-a54c-4184-900a-136a92af01ed"). InnerVolumeSpecName "kube-api-access-n7b2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:49.514889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.513415    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7b2b\" (UniqueName: \"kubernetes.io/projected/3a5fdb64-a54c-4184-900a-136a92af01ed-kube-api-access-n7b2b\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:49.610666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.610501    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" podUID="0b8285bc-132c-40be-924d-c15debf27b9d" containerName="payload-processing" containerID="cri-o://c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c" gracePeriod=25
Nov 01 22:15:49.618788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.617958    5845 generic.go:356] "Generic (PLEG): container finished" podID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerID="c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc" exitCode=0
Nov 01 22:15:49.618788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.618094    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" event={"ID":"3a5fdb64-a54c-4184-900a-136a92af01ed","Type":"ContainerDied","Data":"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"}
Nov 01 22:15:49.618788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.618143    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" event={"ID":"3a5fdb64-a54c-4184-900a-136a92af01ed","Type":"ContainerDied","Data":"6271ad90f26716e10dae3d90579e7095d47ef93ec80e1aaa2e1556b40323d75a"}
Nov 01 22:15:49.618788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.618173    5845 scope.go:122] "RemoveContainer" containerID="c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"
Nov 01 22:15:49.618788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.618397    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"
Nov 01 22:15:49.623732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.623597    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing" containerID="cri-o://8eceb5124b86806a0d35bbaccc52a96ed37d30d085dd09bee571290e4613edd7" gracePeriod=25
Nov 01 22:15:49.676074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.676024    5845 scope.go:122] "RemoveContainer" containerID="c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"
Nov 01 22:15:49.676549 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:49.676526    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc\": container with ID starting with c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc not found: ID does not exist" containerID="c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"
Nov 01 22:15:49.676644 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.676564    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc"} err="failed to get container status \"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc\": rpc error: code = NotFound desc = could not find container \"c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc\": container with ID starting with c93b89d7838f6ee5c74c28ba461662204c16448fa66af8db36cca4240e2acecc not found: ID does not exist"
Nov 01 22:15:49.712530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.712495    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:15:49.716767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.716744    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25"]
Nov 01 22:15:49.857854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.857445    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:15:49.857854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:49.857525    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:15:50.013454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.013419    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:15:50.158427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.131579    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\" (UniqueName: \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\") pod \"0b8285bc-132c-40be-924d-c15debf27b9d\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") "
Nov 01 22:15:50.158427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.131719    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\") pod \"0b8285bc-132c-40be-924d-c15debf27b9d\" (UID: \"0b8285bc-132c-40be-924d-c15debf27b9d\") "
Nov 01 22:15:50.158945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.158901    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume" pod "0b8285bc-132c-40be-924d-c15debf27b9d" (UID: "0b8285bc-132c-40be-924d-c15debf27b9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:15:50.159096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.158942    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz" pod "0b8285bc-132c-40be-924d-c15debf27b9d" (UID: "0b8285bc-132c-40be-924d-c15debf27b9d"). InnerVolumeSpecName "kube-api-access-pm6fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:15:50.164686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.164225    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f2760d-577bdfbb74-6lb25" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.119:50052\" within 1s: context deadline exceeded"
Nov 01 22:15:50.164686 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:15:50.164633    5845 logging.go:55] [core] [Channel #4362 SubChannel #4363]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.119:50052", ServerName: "10.134.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.119:50052: operation was canceled"
Nov 01 22:15:50.235201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.235161    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6fz\" (UniqueName: \"kubernetes.io/projected/0b8285bc-132c-40be-924d-c15debf27b9d-kube-api-access-pm6fz\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:50.235201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.235202    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b8285bc-132c-40be-924d-c15debf27b9d-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:15:50.630484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.630447    5845 generic.go:356] "Generic (PLEG): container finished" podID="0b8285bc-132c-40be-924d-c15debf27b9d" containerID="c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c" exitCode=0
Nov 01 22:15:50.630691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.630530    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"
Nov 01 22:15:50.630691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.630530    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" event={"ID":"0b8285bc-132c-40be-924d-c15debf27b9d","Type":"ContainerDied","Data":"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"}
Nov 01 22:15:50.630691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.630665    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv" event={"ID":"0b8285bc-132c-40be-924d-c15debf27b9d","Type":"ContainerDied","Data":"959c387d7b0d732732ccba1dd5898a91e5289e8b60b3c5c169a0b65650175b46"}
Nov 01 22:15:50.630869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.630697    5845 scope.go:122] "RemoveContainer" containerID="c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"
Nov 01 22:15:50.654508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.654484    5845 scope.go:122] "RemoveContainer" containerID="c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"
Nov 01 22:15:50.654828 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:15:50.654807    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c\": container with ID starting with c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c not found: ID does not exist" containerID="c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"
Nov 01 22:15:50.654908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.654837    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c"} err="failed to get container status \"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c\": rpc error: code = NotFound desc = could not find container \"c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c\": container with ID starting with c4a7d4cfe1c541aa879808c62cf1d91e9c261a11a585c27d5c643bd137e56b5c not found: ID does not exist"
Nov 01 22:15:50.662785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.662756    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:15:50.669125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.669097    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d0a5de-5989495d4b-mfgmv"]
Nov 01 22:15:50.866112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.866071    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8285bc-132c-40be-924d-c15debf27b9d" path="/var/lib/kubelet/pods/0b8285bc-132c-40be-924d-c15debf27b9d/volumes"
Nov 01 22:15:50.866826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:50.866806    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" path="/var/lib/kubelet/pods/3a5fdb64-a54c-4184-900a-136a92af01ed/volumes"
Nov 01 22:15:53.662871 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.662777    5845 generic.go:356] "Generic (PLEG): container finished" podID="9871b3c6-0309-406a-b02b-8696a0daf755" containerID="cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf" exitCode=0
Nov 01 22:15:53.663428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.662863    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerDied","Data":"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"}
Nov 01 22:15:53.663428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.663412    5845 scope.go:122] "RemoveContainer" containerID="cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"
Nov 01 22:15:53.665577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.665536    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:15:53.665696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.665622    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:15:53.665696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:53.665656    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:15:54.703324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:54.703149    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerStarted","Data":"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"}
Nov 01 22:15:54.991207 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:54.990733    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:15:54.991622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:54.991449    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:54.991622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:54.991528    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:55.969324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:55.968393    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:55.969324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:55.968451    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:56.968849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:56.968810    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:56.969058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:56.968888    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:57.964741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:57.964694    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:57.965238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:57.964787    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:58.973321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:58.972367    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:58.973321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:58.972438    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:59.379382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:59.379347    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:15:59.962322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:59.959127    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:15:59.962322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:59.959235    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 01 22:15:59.995747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:15:59.995712    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:16:00.964128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:00.964091    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:16:00.964558 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:00.964534    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:16:00.965308 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:00.965281    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:16:04.823812 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:04.823774    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:16:04.996537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:04.996501    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing" containerID="cri-o://cfd00ffddf63c3e70931f717e5c4b4bdcd840af9a04819c6e3c672b5e2a56e09" gracePeriod=25
Nov 01 22:16:05.020986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:05.020958    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:16:05.791427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:05.791372    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:16:05.801966 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:05.801934    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:16:08.812019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:08.811984    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:09.378944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:09.378901    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:09.908082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:09.908054    5845 generic.go:356] "Generic (PLEG): container finished" podID="1428094a-6382-4d52-8786-da737f78c348" containerID="8eceb5124b86806a0d35bbaccc52a96ed37d30d085dd09bee571290e4613edd7" exitCode=0
Nov 01 22:16:09.908473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:09.908134    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" event={"ID":"1428094a-6382-4d52-8786-da737f78c348","Type":"ContainerDied","Data":"8eceb5124b86806a0d35bbaccc52a96ed37d30d085dd09bee571290e4613edd7"}
Nov 01 22:16:09.941009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:09.940988    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:16:10.015291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.015252    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\") pod \"1428094a-6382-4d52-8786-da737f78c348\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") "
Nov 01 22:16:10.015516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.015417    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\" (UniqueName: \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\") pod \"1428094a-6382-4d52-8786-da737f78c348\" (UID: \"1428094a-6382-4d52-8786-da737f78c348\") "
Nov 01 22:16:10.015752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.015715    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume" pod "1428094a-6382-4d52-8786-da737f78c348" (UID: "1428094a-6382-4d52-8786-da737f78c348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:10.016811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.016785    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428094a-6382-4d52-8786-da737f78c348-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:10.018706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.018682    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5" pod "1428094a-6382-4d52-8786-da737f78c348" (UID: "1428094a-6382-4d52-8786-da737f78c348"). InnerVolumeSpecName "kube-api-access-2cfh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:10.021807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.021766    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" podUID="bf9f93d5-7660-4fd1-8340-fca5525292dd" containerName="payload-processing" containerID="cri-o://ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2" gracePeriod=25
Nov 01 22:16:10.118771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.118743    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cfh5\" (UniqueName: \"kubernetes.io/projected/1428094a-6382-4d52-8786-da737f78c348-kube-api-access-2cfh5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:10.361596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.361570    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:16:10.425314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.425268    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\") pod \"bf9f93d5-7660-4fd1-8340-fca5525292dd\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") "
Nov 01 22:16:10.425472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.425412    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\" (UniqueName: \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\") pod \"bf9f93d5-7660-4fd1-8340-fca5525292dd\" (UID: \"bf9f93d5-7660-4fd1-8340-fca5525292dd\") "
Nov 01 22:16:10.425646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.425620    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume" pod "bf9f93d5-7660-4fd1-8340-fca5525292dd" (UID: "bf9f93d5-7660-4fd1-8340-fca5525292dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:10.426085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.426062    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9f93d5-7660-4fd1-8340-fca5525292dd-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:10.427639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.427613    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd" pod "bf9f93d5-7660-4fd1-8340-fca5525292dd" (UID: "bf9f93d5-7660-4fd1-8340-fca5525292dd"). InnerVolumeSpecName "kube-api-access-2n5gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:10.527452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.527376    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5gd\" (UniqueName: \"kubernetes.io/projected/bf9f93d5-7660-4fd1-8340-fca5525292dd-kube-api-access-2n5gd\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:10.792603 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.792515    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing" containerID="cri-o://47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896" gracePeriod=25
Nov 01 22:16:10.803196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.803149    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing" containerID="cri-o://18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6" gracePeriod=25
Nov 01 22:16:10.917080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.917055    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"
Nov 01 22:16:10.917457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.917096    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl" event={"ID":"1428094a-6382-4d52-8786-da737f78c348","Type":"ContainerDied","Data":"1f6340c3c5a7182d33e5e6dac8f8b599f4cd209188f9e8becf278b5942b5c443"}
Nov 01 22:16:10.917457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.917152    5845 scope.go:122] "RemoveContainer" containerID="8eceb5124b86806a0d35bbaccc52a96ed37d30d085dd09bee571290e4613edd7"
Nov 01 22:16:10.918741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.918717    5845 generic.go:356] "Generic (PLEG): container finished" podID="bf9f93d5-7660-4fd1-8340-fca5525292dd" containerID="ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2" exitCode=0
Nov 01 22:16:10.918844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.918753    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"
Nov 01 22:16:10.918904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.918838    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" event={"ID":"bf9f93d5-7660-4fd1-8340-fca5525292dd","Type":"ContainerDied","Data":"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"}
Nov 01 22:16:10.918965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.918903    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh" event={"ID":"bf9f93d5-7660-4fd1-8340-fca5525292dd","Type":"ContainerDied","Data":"8005befb9d7705b44dcf5cce5d98b462245342562a4274a250dca5ebe2fa2453"}
Nov 01 22:16:10.929535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.929519    5845 scope.go:122] "RemoveContainer" containerID="ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"
Nov 01 22:16:10.942006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.941985    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:16:10.942220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.942195    5845 scope.go:122] "RemoveContainer" containerID="ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"
Nov 01 22:16:10.942584 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:10.942556    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2\": container with ID starting with ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2 not found: ID does not exist" containerID="ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"
Nov 01 22:16:10.942707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.942592    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2"} err="failed to get container status \"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2\": rpc error: code = NotFound desc = could not find container \"ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2\": container with ID starting with ae6b3433eec110fa66c36a3254223157270e58dfae46a482f33ade98f8328eb2 not found: ID does not exist"
Nov 01 22:16:10.946572 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.946552    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0a5de-7fbff57d8c-zz2kl"]
Nov 01 22:16:10.957403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.957338    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:16:10.961065 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:10.961041    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9d8b5d-76d68746b6-7s7mh"]
Nov 01 22:16:12.864511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:12.864480    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1428094a-6382-4d52-8786-da737f78c348" path="/var/lib/kubelet/pods/1428094a-6382-4d52-8786-da737f78c348/volumes"
Nov 01 22:16:12.865018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:12.865002    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9f93d5-7660-4fd1-8340-fca5525292dd" path="/var/lib/kubelet/pods/bf9f93d5-7660-4fd1-8340-fca5525292dd/volumes"
Nov 01 22:16:13.997035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:13.997000    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:13.997472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:13.997045    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:15.276846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:15.276814    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:16:15.277821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:15.277029    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" containerID="cri-o://0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa" gracePeriod=30
Nov 01 22:16:18.825963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:18.825917    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:19.240517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.240482    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:16:19.241021 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.240961    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" podUID="9e568845-e244-4484-a952-876893d3f477" containerName="maas-api" containerID="cri-o://d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c" gracePeriod=30
Nov 01 22:16:19.492700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.492625    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:16:19.493203 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.493159    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" podUID="58029f31-95cf-4ff1-8ba9-ac769afa0833" containerName="maas-api" containerID="cri-o://3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89" gracePeriod=30
Nov 01 22:16:19.617082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.617042    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:16:19.617764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.617116    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:16:19.678077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.678048    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:16:19.871348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.861093    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\") pod \"9e568845-e244-4484-a952-876893d3f477\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") "
Nov 01 22:16:19.871348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.861160    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\" (UniqueName: \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\") pod \"9e568845-e244-4484-a952-876893d3f477\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") "
Nov 01 22:16:19.871348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.861265    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\") pod \"9e568845-e244-4484-a952-876893d3f477\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") "
Nov 01 22:16:19.871348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.861458    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") pod \"9e568845-e244-4484-a952-876893d3f477\" (UID: \"9e568845-e244-4484-a952-876893d3f477\") "
Nov 01 22:16:19.970345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.934809    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca" pod "9e568845-e244-4484-a952-876893d3f477" (UID: "9e568845-e244-4484-a952-876893d3f477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:19.990324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.986211    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e568845-e244-4484-a952-876893d3f477-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:19.990324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:19.987480    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.207785    5845 generic.go:356] "Generic (PLEG): container finished" podID="58029f31-95cf-4ff1-8ba9-ac769afa0833" containerID="3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89" exitCode=0
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.207903    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" event={"ID":"58029f31-95cf-4ff1-8ba9-ac769afa0833","Type":"ContainerDied","Data":"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"}
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.207954    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr" event={"ID":"58029f31-95cf-4ff1-8ba9-ac769afa0833","Type":"ContainerDied","Data":"9af97aaf8ef380cecab0134c52298c8b6efef2073edac3ea264cb2cce176f93f"}
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.207984    5845 scope.go:122] "RemoveContainer" containerID="3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.208139    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.209129    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\" (UniqueName: \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\") pod \"58029f31-95cf-4ff1-8ba9-ac769afa0833\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") "
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.209333    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") pod \"58029f31-95cf-4ff1-8ba9-ac769afa0833\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") "
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.209542    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\") pod \"58029f31-95cf-4ff1-8ba9-ac769afa0833\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") "
Nov 01 22:16:20.211012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.209664    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\") pod \"58029f31-95cf-4ff1-8ba9-ac769afa0833\" (UID: \"58029f31-95cf-4ff1-8ba9-ac769afa0833\") "
Nov 01 22:16:20.212106 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.211997    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca" pod "58029f31-95cf-4ff1-8ba9-ac769afa0833" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:20.219791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.218886    5845 generic.go:356] "Generic (PLEG): container finished" podID="9e568845-e244-4484-a952-876893d3f477" containerID="d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c" exitCode=0
Nov 01 22:16:20.219791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.218978    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" event={"ID":"9e568845-e244-4484-a952-876893d3f477","Type":"ContainerDied","Data":"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"}
Nov 01 22:16:20.219791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.219025    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z" event={"ID":"9e568845-e244-4484-a952-876893d3f477","Type":"ContainerDied","Data":"808918f0ed289ce8e030d648f9818eea5bf716f7343b86e828260ddd6df15539"}
Nov 01 22:16:20.219791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.219101    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"
Nov 01 22:16:20.219791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.219749    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp" pod "9e568845-e244-4484-a952-876893d3f477" (UID: "9e568845-e244-4484-a952-876893d3f477"). InnerVolumeSpecName "kube-api-access-sw8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:20.255029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.253482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs" pod "58029f31-95cf-4ff1-8ba9-ac769afa0833" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:16:20.255029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.254387    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls" pod "58029f31-95cf-4ff1-8ba9-ac769afa0833" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:16:20.255029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.254460    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls" pod "9e568845-e244-4484-a952-876893d3f477" (UID: "9e568845-e244-4484-a952-876893d3f477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:16:20.256250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.256200    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs" pod "9e568845-e244-4484-a952-876893d3f477" (UID: "9e568845-e244-4484-a952-876893d3f477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:16:20.256575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.256541    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8" pod "58029f31-95cf-4ff1-8ba9-ac769afa0833" (UID: "58029f31-95cf-4ff1-8ba9-ac769afa0833"). InnerVolumeSpecName "kube-api-access-djmh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321400    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321439    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321454    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58029f31-95cf-4ff1-8ba9-ac769afa0833-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321470    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmh8\" (UniqueName: \"kubernetes.io/projected/58029f31-95cf-4ff1-8ba9-ac769afa0833-kube-api-access-djmh8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321484    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e568845-e244-4484-a952-876893d3f477-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321501    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8dp\" (UniqueName: \"kubernetes.io/projected/9e568845-e244-4484-a952-876893d3f477-kube-api-access-sw8dp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.323180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.321516    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58029f31-95cf-4ff1-8ba9-ac769afa0833-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:20.334962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.334740    5845 scope.go:122] "RemoveContainer" containerID="3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"
Nov 01 22:16:20.335216 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:20.335186    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89\": container with ID starting with 3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89 not found: ID does not exist" containerID="3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"
Nov 01 22:16:20.335311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.335229    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89"} err="failed to get container status \"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89\": rpc error: code = NotFound desc = could not find container \"3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89\": container with ID starting with 3c6a987cb8b0624523e1a670ce4b37848580cd127d38bbab853dafb7034dbb89 not found: ID does not exist"
Nov 01 22:16:20.335311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.335255    5845 scope.go:122] "RemoveContainer" containerID="d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"
Nov 01 22:16:20.438132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.437476    5845 scope.go:122] "RemoveContainer" containerID="d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"
Nov 01 22:16:20.455141 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:20.455098    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c\": container with ID starting with d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c not found: ID does not exist" containerID="d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"
Nov 01 22:16:20.455307 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.455149    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c"} err="failed to get container status \"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c\": rpc error: code = NotFound desc = could not find container \"d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c\": container with ID starting with d70f3aae75f8c5e244b590951680e0c214f3ee1806965cebba8f032141bdc67c not found: ID does not exist"
Nov 01 22:16:20.548400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.548368    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:16:20.556143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.556102    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8483c2-548854584c-rjfhr"]
Nov 01 22:16:20.574431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.574400    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:16:20.581665 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.581637    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17957f-7d5d96f8ff-4k56z"]
Nov 01 22:16:20.914812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.914784    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58029f31-95cf-4ff1-8ba9-ac769afa0833" path="/var/lib/kubelet/pods/58029f31-95cf-4ff1-8ba9-ac769afa0833/volumes"
Nov 01 22:16:20.915322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.915291    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:16:20.915431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:20.915413    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e568845-e244-4484-a952-876893d3f477" path="/var/lib/kubelet/pods/9e568845-e244-4484-a952-876893d3f477/volumes"
Nov 01 22:16:21.017706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.017664    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.017904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.017826    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.017986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.017969    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018203    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:21.018424 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018405    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018478    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018533    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018556    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018625    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.018903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.018666    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\") pod \"22a04513-864a-4c2f-be26-be04673400a7\" (UID: \"22a04513-864a-4c2f-be26-be04673400a7\") "
Nov 01 22:16:21.019107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019081    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:21.019222 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019200    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:21.019449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019428    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019571    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019610    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019623    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a04513-864a-4c2f-be26-be04673400a7-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019640    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.019649    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.020408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.020144    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:21.020722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.020509    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:21.020848 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.020831    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:21.021247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.021227    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6" pod "22a04513-864a-4c2f-be26-be04673400a7" (UID: "22a04513-864a-4c2f-be26-be04673400a7"). InnerVolumeSpecName "kube-api-access-dg4l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:21.121385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.121353    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a04513-864a-4c2f-be26-be04673400a7-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.121385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.121380    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.121385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.121391    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4l6\" (UniqueName: \"kubernetes.io/projected/22a04513-864a-4c2f-be26-be04673400a7-kube-api-access-dg4l6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.121648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.121401    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.121648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.121409    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a04513-864a-4c2f-be26-be04673400a7-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:21.233408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.233375    5845 generic.go:356] "Generic (PLEG): container finished" podID="22a04513-864a-4c2f-be26-be04673400a7" containerID="0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa" exitCode=0
Nov 01 22:16:21.233569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.233433    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" event={"ID":"22a04513-864a-4c2f-be26-be04673400a7","Type":"ContainerDied","Data":"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"}
Nov 01 22:16:21.233569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.233450    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"
Nov 01 22:16:21.233569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.233498    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt" event={"ID":"22a04513-864a-4c2f-be26-be04673400a7","Type":"ContainerDied","Data":"6304d10e388b6d403265eb7d3288e5227140f82367a01a57ba3b3f3b57e2bc09"}
Nov 01 22:16:21.233569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.233516    5845 scope.go:122] "RemoveContainer" containerID="0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"
Nov 01 22:16:21.246745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.246726    5845 scope.go:122] "RemoveContainer" containerID="0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"
Nov 01 22:16:21.246987 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:21.246965    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa\": container with ID starting with 0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa not found: ID does not exist" containerID="0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"
Nov 01 22:16:21.247082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.246999    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa"} err="failed to get container status \"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa\": rpc error: code = NotFound desc = could not find container \"0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa\": container with ID starting with 0c05988d9a18a3ff9398634bf6536244608164383f7dd578fcfb471e4626dbfa not found: ID does not exist"
Nov 01 22:16:21.269190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.269164    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:16:21.275727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:21.275695    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-gwpjt"]
Nov 01 22:16:22.904607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:22.904568    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a04513-864a-4c2f-be26-be04673400a7" path="/var/lib/kubelet/pods/22a04513-864a-4c2f-be26-be04673400a7/volumes"
Nov 01 22:16:23.997507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:23.997473    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:23.998267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:23.998235    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:24.404459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:24.404385    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 22:16:24.404600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:24.404578    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy" containerID="cri-o://aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a" gracePeriod=30
Nov 01 22:16:25.412498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.412405    5845 generic.go:356] "Generic (PLEG): container finished" podID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerID="cfd00ffddf63c3e70931f717e5c4b4bdcd840af9a04819c6e3c672b5e2a56e09" exitCode=0
Nov 01 22:16:25.413267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.413243    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" event={"ID":"c3378d78-f15c-4d7b-9efe-352664f6a03b","Type":"ContainerDied","Data":"cfd00ffddf63c3e70931f717e5c4b4bdcd840af9a04819c6e3c672b5e2a56e09"}
Nov 01 22:16:25.470824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.470790    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:16:25.535979 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.535945    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\" (UniqueName: \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\") pod \"c3378d78-f15c-4d7b-9efe-352664f6a03b\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") "
Nov 01 22:16:25.536181 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.536092    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\") pod \"c3378d78-f15c-4d7b-9efe-352664f6a03b\" (UID: \"c3378d78-f15c-4d7b-9efe-352664f6a03b\") "
Nov 01 22:16:25.536498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.536468    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume" pod "c3378d78-f15c-4d7b-9efe-352664f6a03b" (UID: "c3378d78-f15c-4d7b-9efe-352664f6a03b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:25.537120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.537025    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3378d78-f15c-4d7b-9efe-352664f6a03b-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:25.538741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.538718    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts" pod "c3378d78-f15c-4d7b-9efe-352664f6a03b" (UID: "c3378d78-f15c-4d7b-9efe-352664f6a03b"). InnerVolumeSpecName "kube-api-access-nh9ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:25.639231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:25.639197    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh9ts\" (UniqueName: \"kubernetes.io/projected/c3378d78-f15c-4d7b-9efe-352664f6a03b-kube-api-access-nh9ts\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:26.422355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.422285    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"
Nov 01 22:16:26.422355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.422346    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj" event={"ID":"c3378d78-f15c-4d7b-9efe-352664f6a03b","Type":"ContainerDied","Data":"ece199ceb2e48b674ca695cefd2cf6266842b971c4d53ac0d02e20a27bd9a2cf"}
Nov 01 22:16:26.422806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.422410    5845 scope.go:122] "RemoveContainer" containerID="cfd00ffddf63c3e70931f717e5c4b4bdcd840af9a04819c6e3c672b5e2a56e09"
Nov 01 22:16:26.473819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.473785    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:16:26.478390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.478359    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17957f-86785bdb55-vvrzj"]
Nov 01 22:16:26.897884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:26.897845    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" path="/var/lib/kubelet/pods/c3378d78-f15c-4d7b-9efe-352664f6a03b/volumes"
Nov 01 22:16:28.124756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.122805    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:16:28.124756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.123086    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy" containerID="cri-o://8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f" gracePeriod=30
Nov 01 22:16:28.193324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.173624    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:16:28.193324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.173938    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy" containerID="cri-o://579a2e7ee54d2ddad0e3f2d5027e4670a06799ee3bf3116c8a39eb52e38af40c" gracePeriod=30
Nov 01 22:16:28.282004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.269847    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:16:28.282004 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:28.270119    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" containerID="cri-o://7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236" gracePeriod=30
Nov 01 22:16:29.745916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.745648    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 22:16:29.937879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.937848    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.937945    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938034    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938120    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938193    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938269    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938324    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938455    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.938559 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.938544    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\") pod \"2082417d-f041-4d19-84b5-1161d41cddd1\" (UID: \"2082417d-f041-4d19-84b5-1161d41cddd1\") "
Nov 01 22:16:29.939351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.939145    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:29.939817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.939788    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:29.939969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.939844    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:29.940634 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.940611    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:29.940740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.940715    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:29.942445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.942419    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:29.944435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.944409    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:29.944547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.944443    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:29.944609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:29.944560    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv" pod "2082417d-f041-4d19-84b5-1161d41cddd1" (UID: "2082417d-f041-4d19-84b5-1161d41cddd1"). InnerVolumeSpecName "kube-api-access-nldrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:30.030190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.030155    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:16:30.031667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031635    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0b8285bc-132c-40be-924d-c15debf27b9d" containerName="payload-processing"
Nov 01 22:16:30.031667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031665    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8285bc-132c-40be-924d-c15debf27b9d" containerName="payload-processing"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031683    5845 cpu_manager.go:414] "containerMap: removing container" podUID="58029f31-95cf-4ff1-8ba9-ac769afa0833" containerName="maas-api"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031691    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58029f31-95cf-4ff1-8ba9-ac769afa0833" containerName="maas-api"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031707    5845 cpu_manager.go:414] "containerMap: removing container" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031714    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031724    5845 cpu_manager.go:414] "containerMap: removing container" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031731    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031744    5845 cpu_manager.go:414] "containerMap: removing container" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031750    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031769    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031776    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031788    5845 cpu_manager.go:414] "containerMap: removing container" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031795    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031810    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.031817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031819    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031829    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9e568845-e244-4484-a952-876893d3f477" containerName="maas-api"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031836    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e568845-e244-4484-a952-876893d3f477" containerName="maas-api"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031856    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031863    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031886    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031893    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031907    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031915    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031931    5845 cpu_manager.go:414] "containerMap: removing container" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031938    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031949    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031957    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031970    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031977    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031989    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.031996    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032010    5845 cpu_manager.go:414] "containerMap: removing container" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032017    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032030    5845 cpu_manager.go:414] "containerMap: removing container" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032037    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032059    5845 cpu_manager.go:414] "containerMap: removing container" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032066    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032093    5845 cpu_manager.go:414] "containerMap: removing container" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032099    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032113    5845 cpu_manager.go:414] "containerMap: removing container" podUID="bf9f93d5-7660-4fd1-8340-fca5525292dd" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032121    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9f93d5-7660-4fd1-8340-fca5525292dd" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032134    5845 cpu_manager.go:414] "containerMap: removing container" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032141    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032156    5845 cpu_manager.go:414] "containerMap: removing container" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032163    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032179    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032187    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032418    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5fdb64-a54c-4184-900a-136a92af01ed" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032443    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032453    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3378d78-f15c-4d7b-9efe-352664f6a03b" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032467    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032476    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032486    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032494    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1def6a01-5d2b-4c1b-b3e9-4d7092975fce" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032505    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032515    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a04513-864a-4c2f-be26-be04673400a7" containerName="istio-proxy"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032532    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032542    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032553    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032562    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032573    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9f93d5-7660-4fd1-8340-fca5525292dd" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032581    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032592    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a9e037-1694-474f-9acf-e71d4963930b" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032602    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1428094a-6382-4d52-8786-da737f78c348" containerName="payload-pre-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032615    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8285bc-132c-40be-924d-c15debf27b9d" containerName="payload-processing"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032625    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="58029f31-95cf-4ff1-8ba9-ac769afa0833" containerName="maas-api"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032634    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032644    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032652    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032662    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" containerName="istio-proxy"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032673    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8f0ea5-3f87-4ced-a26a-eeed9a7e1446" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032684    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:30.033278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.032694    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e568845-e244-4484-a952-876893d3f477" containerName="maas-api"
Nov 01 22:16:30.037525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.037500    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.056977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.056936    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:16:30.075856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.075823    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e3890e-openshift-default-dockercfg-j98nz\""
Nov 01 22:16:30.076103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.075963    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076033    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076101    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp94l\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076139    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076223    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076393    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076481    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.076587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076561    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.076958    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077207    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077226    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077240    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2082417d-f041-4d19-84b5-1161d41cddd1-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077260    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077273    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077288    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2082417d-f041-4d19-84b5-1161d41cddd1-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077321    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldrv\" (UniqueName: \"kubernetes.io/projected/2082417d-f041-4d19-84b5-1161d41cddd1-kube-api-access-nldrv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077338    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.080347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.077352    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2082417d-f041-4d19-84b5-1161d41cddd1-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:30.183709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.183667    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.201714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.201681    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.202054 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.202030    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.202228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.202210    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp94l\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.203785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.203762    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.214360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.203999    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.215101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.215074    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.215287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.215264    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.215473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.215452    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.216232 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.216206    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.216429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.203334    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.216811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.216785    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.225317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.195168    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.225317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.219432    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.285379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.271199    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.311533 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.307462    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.313193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.312236    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.313193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.312953    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp94l\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.394980 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.394948    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:30.494586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.494153    5845 generic.go:356] "Generic (PLEG): container finished" podID="2082417d-f041-4d19-84b5-1161d41cddd1" containerID="aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a" exitCode=0
Nov 01 22:16:30.494586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.494213    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"
Nov 01 22:16:30.494586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.494368    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" event={"ID":"2082417d-f041-4d19-84b5-1161d41cddd1","Type":"ContainerDied","Data":"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"}
Nov 01 22:16:30.494586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.494428    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d" event={"ID":"2082417d-f041-4d19-84b5-1161d41cddd1","Type":"ContainerDied","Data":"b04ee006d57452ea7d6c2ae9797887118ccafc3a9c9e85654eaa576d6599c35f"}
Nov 01 22:16:30.494586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.494458    5845 scope.go:122] "RemoveContainer" containerID="aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"
Nov 01 22:16:30.519746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.519621    5845 scope.go:122] "RemoveContainer" containerID="aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"
Nov 01 22:16:30.521195 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:30.519979    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a\": container with ID starting with aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a not found: ID does not exist" containerID="aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"
Nov 01 22:16:30.521195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.520058    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a"} err="failed to get container status \"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a\": rpc error: code = NotFound desc = could not find container \"aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a\": container with ID starting with aa4f30ec7d8f79ce21eb83309cc3cdd099d253e9f104f4bb55ef988a8901a34a not found: ID does not exist"
Nov 01 22:16:30.534587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.534563    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 22:16:30.541182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.541163    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-7bg9d"]
Nov 01 22:16:30.709779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.709745    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:16:30.710971 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:16:30.710934    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c66cc6_05aa_449e_aa44_618406f83882.slice/crio-85e870bca09b2d48bcf4f36068bb7f8da8884f0495e589bb25b63f8758f71748 WatchSource:0}: Error finding container 85e870bca09b2d48bcf4f36068bb7f8da8884f0495e589bb25b63f8758f71748: Status 404 returned error can't find the container with id 85e870bca09b2d48bcf4f36068bb7f8da8884f0495e589bb25b63f8758f71748
Nov 01 22:16:30.715009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.714966    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:30.715134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.715082    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:30.715134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.715121    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:30.898843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:30.898759    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2082417d-f041-4d19-84b5-1161d41cddd1" path="/var/lib/kubelet/pods/2082417d-f041-4d19-84b5-1161d41cddd1/volumes"
Nov 01 22:16:31.128193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.128157    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:16:31.224028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.223667    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:16:31.255883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.255800    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\") pod \"339a5d91-271c-4509-b485-908329cdc38c\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") "
Nov 01 22:16:31.255883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.255847    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\") pod \"339a5d91-271c-4509-b485-908329cdc38c\" (UID: \"339a5d91-271c-4509-b485-908329cdc38c\") "
Nov 01 22:16:31.256815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.256784    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume" pod "339a5d91-271c-4509-b485-908329cdc38c" (UID: "339a5d91-271c-4509-b485-908329cdc38c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:31.258978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.258947    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2" pod "339a5d91-271c-4509-b485-908329cdc38c" (UID: "339a5d91-271c-4509-b485-908329cdc38c"). InnerVolumeSpecName "kube-api-access-jpmz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:31.358185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.358153    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\" (UniqueName: \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\") pod \"37272cfa-7de1-4bdf-bd98-2ded556f162a\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") "
Nov 01 22:16:31.358440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.358246    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\") pod \"37272cfa-7de1-4bdf-bd98-2ded556f162a\" (UID: \"37272cfa-7de1-4bdf-bd98-2ded556f162a\") "
Nov 01 22:16:31.358703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.358676    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume" pod "37272cfa-7de1-4bdf-bd98-2ded556f162a" (UID: "37272cfa-7de1-4bdf-bd98-2ded556f162a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:31.359020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.359002    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpmz2\" (UniqueName: \"kubernetes.io/projected/339a5d91-271c-4509-b485-908329cdc38c-kube-api-access-jpmz2\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:31.359086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.359025    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339a5d91-271c-4509-b485-908329cdc38c-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:31.359086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.359041    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37272cfa-7de1-4bdf-bd98-2ded556f162a-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:31.360478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.360458    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg" pod "37272cfa-7de1-4bdf-bd98-2ded556f162a" (UID: "37272cfa-7de1-4bdf-bd98-2ded556f162a"). InnerVolumeSpecName "kube-api-access-c9pfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:31.461159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.461125    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pfg\" (UniqueName: \"kubernetes.io/projected/37272cfa-7de1-4bdf-bd98-2ded556f162a-kube-api-access-c9pfg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:31.506266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.506175    5845 generic.go:356] "Generic (PLEG): container finished" podID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerID="18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6" exitCode=0
Nov 01 22:16:31.506266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.506234    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"
Nov 01 22:16:31.506515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.506260    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" event={"ID":"37272cfa-7de1-4bdf-bd98-2ded556f162a","Type":"ContainerDied","Data":"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"}
Nov 01 22:16:31.506515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.506365    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw" event={"ID":"37272cfa-7de1-4bdf-bd98-2ded556f162a","Type":"ContainerDied","Data":"2e1ea825951a68589f24e86747b0c8b7a598258f102f8e2d4e065e88dee16619"}
Nov 01 22:16:31.506515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.506405    5845 scope.go:122] "RemoveContainer" containerID="18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"
Nov 01 22:16:31.507974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.507949    5845 generic.go:356] "Generic (PLEG): container finished" podID="339a5d91-271c-4509-b485-908329cdc38c" containerID="47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896" exitCode=0
Nov 01 22:16:31.508099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.508007    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"
Nov 01 22:16:31.508099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.508014    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" event={"ID":"339a5d91-271c-4509-b485-908329cdc38c","Type":"ContainerDied","Data":"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"}
Nov 01 22:16:31.508099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.508059    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92" event={"ID":"339a5d91-271c-4509-b485-908329cdc38c","Type":"ContainerDied","Data":"e20f361da7aa39250bd2ea1893813af0a09480094cbce80f00f28f46ab8b24bb"}
Nov 01 22:16:31.510827 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.510778    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerStarted","Data":"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"}
Nov 01 22:16:31.510929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.510828    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerStarted","Data":"85e870bca09b2d48bcf4f36068bb7f8da8884f0495e589bb25b63f8758f71748"}
Nov 01 22:16:31.530544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.530521    5845 scope.go:122] "RemoveContainer" containerID="18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"
Nov 01 22:16:31.531083 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:31.531055    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6\": container with ID starting with 18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6 not found: ID does not exist" containerID="18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"
Nov 01 22:16:31.531191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.531096    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6"} err="failed to get container status \"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6\": rpc error: code = NotFound desc = could not find container \"18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6\": container with ID starting with 18898be08b2d682d88af0adc51b8d1679f539844d9c8b169fccd6c9cd75d06f6 not found: ID does not exist"
Nov 01 22:16:31.531191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.531119    5845 scope.go:122] "RemoveContainer" containerID="47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"
Nov 01 22:16:31.546239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.546168    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podStartSLOduration=1.546153654 podStartE2EDuration="1.546153654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:31.54380301 +0000 UTC m=+1894.799982774" watchObservedRunningTime="2026-11-01 22:16:31.546153654 +0000 UTC m=+1894.802333418"
Nov 01 22:16:31.546696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.546676    5845 scope.go:122] "RemoveContainer" containerID="47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"
Nov 01 22:16:31.547061 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:31.547033    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896\": container with ID starting with 47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896 not found: ID does not exist" containerID="47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"
Nov 01 22:16:31.547157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.547068    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896"} err="failed to get container status \"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896\": rpc error: code = NotFound desc = could not find container \"47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896\": container with ID starting with 47713c79ff9c8497234d63fb370293ab19d96640d76bfb31ff06fa3db8ded896 not found: ID does not exist"
Nov 01 22:16:31.565174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.565144    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:16:31.569488 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.569455    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8483c2-57fd7d7846-mgl92"]
Nov 01 22:16:31.585583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.585552    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:16:31.590345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:31.590314    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8483c2-79469b67b7-s88jw"]
Nov 01 22:16:32.395833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.395797    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:32.401566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.401543    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:32.523360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.523330    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:32.524555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.524528    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:16:32.866168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.866135    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="339a5d91-271c-4509-b485-908329cdc38c" path="/var/lib/kubelet/pods/339a5d91-271c-4509-b485-908329cdc38c/volumes"
Nov 01 22:16:32.866609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:32.866594    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" path="/var/lib/kubelet/pods/37272cfa-7de1-4bdf-bd98-2ded556f162a/volumes"
Nov 01 22:16:33.539907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.539869    5845 generic.go:356] "Generic (PLEG): container finished" podID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerID="579a2e7ee54d2ddad0e3f2d5027e4670a06799ee3bf3116c8a39eb52e38af40c" exitCode=0
Nov 01 22:16:33.541238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.541203    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" event={"ID":"99efaa10-3ca4-4a9d-98f0-d776ffb1a467","Type":"ContainerDied","Data":"579a2e7ee54d2ddad0e3f2d5027e4670a06799ee3bf3116c8a39eb52e38af40c"}
Nov 01 22:16:33.662448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.662411    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:16:33.713667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713597    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.713801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713743    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.713801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713781    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713927    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713927    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.713959    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.714001    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.714028    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.714152    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.714232    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\") pod \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\" (UID: \"99efaa10-3ca4-4a9d-98f0-d776ffb1a467\") "
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.714893    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.715102    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.715350    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.715597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.715411    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.716441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.716396    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:33.720379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.720354    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.723220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.723191    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "kube-api-access-v4nlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.723416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.723397    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:33.724033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.724010    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy" pod "99efaa10-3ca4-4a9d-98f0-d776ffb1a467" (UID: "99efaa10-3ca4-4a9d-98f0-d776ffb1a467"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.799349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.788428    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:16:33.815534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815501    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815691    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815753    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815775    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815829    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815882    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.815936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815920    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.816044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.815966    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.816044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816014    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\") pod \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\" (UID: \"ab0c0a23-451c-43e2-a709-d123d0dd63fb\") "
Nov 01 22:16:33.816584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816560    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.816833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816811    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816834    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816852    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816866    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816878    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816892    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.816900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816900    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.817213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816909    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.817213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816918    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.817213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.816927    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nlg\" (UniqueName: \"kubernetes.io/projected/99efaa10-3ca4-4a9d-98f0-d776ffb1a467-kube-api-access-v4nlg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.817213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.817028    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.817526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.817504    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:33.817722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.817699    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.823148 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.823114    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.823520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.823499    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.825545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.825519    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:33.826195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.826169    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc" pod "ab0c0a23-451c-43e2-a709-d123d0dd63fb" (UID: "ab0c0a23-451c-43e2-a709-d123d0dd63fb"). InnerVolumeSpecName "kube-api-access-vrxlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.839672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.839647    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:16:33.917880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.917847    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.917899    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.917962    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918028    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918128    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918203    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918289    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918376    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918413    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\") pod \"f303a365-4e50-4f59-84fe-f6e83ca4c765\" (UID: \"f303a365-4e50-4f59-84fe-f6e83ca4c765\") "
Nov 01 22:16:33.918994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.918874    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.919740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.919112    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.919740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.919402    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.919740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.919482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:33.919740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.919522    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:33.922850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.922825    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.922981    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923003    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923018    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923035    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923048    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923061    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923076    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxlc\" (UniqueName: \"kubernetes.io/projected/ab0c0a23-451c-43e2-a709-d123d0dd63fb-kube-api-access-vrxlc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923091    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923105    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923118    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f303a365-4e50-4f59-84fe-f6e83ca4c765-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923133    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c0a23-451c-43e2-a709-d123d0dd63fb-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923147    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0c0a23-451c-43e2-a709-d123d0dd63fb-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:33.923443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923198    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.923726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923478    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:33.923726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.923700    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "kube-api-access-kzjvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:33.924522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:33.924495    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy" pod "f303a365-4e50-4f59-84fe-f6e83ca4c765" (UID: "f303a365-4e50-4f59-84fe-f6e83ca4c765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:34.025089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.025058    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:34.025089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.025084    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:34.025089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.025093    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:34.025371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.025106    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjvz\" (UniqueName: \"kubernetes.io/projected/f303a365-4e50-4f59-84fe-f6e83ca4c765-kube-api-access-kzjvz\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:34.548706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.548673    5845 generic.go:356] "Generic (PLEG): container finished" podID="3f10c1d2-c233-4174-939b-676ec58c594d" containerID="4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701" exitCode=0
Nov 01 22:16:34.549183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.548752    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerDied","Data":"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"}
Nov 01 22:16:34.549258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.549224    5845 scope.go:122] "RemoveContainer" containerID="4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"
Nov 01 22:16:34.550331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.550288    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd" event={"ID":"99efaa10-3ca4-4a9d-98f0-d776ffb1a467","Type":"ContainerDied","Data":"feee922654db1305418006ca0b3f0cd496767ca4dda7b8d5b8fbdd803d9530af"}
Nov 01 22:16:34.550473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.550333    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"
Nov 01 22:16:34.550473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.550367    5845 scope.go:122] "RemoveContainer" containerID="579a2e7ee54d2ddad0e3f2d5027e4670a06799ee3bf3116c8a39eb52e38af40c"
Nov 01 22:16:34.551248 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.551218    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:34.551406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.551313    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:34.551406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.551363    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:16:34.552815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.551889    5845 generic.go:356] "Generic (PLEG): container finished" podID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerID="7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236" exitCode=0
Nov 01 22:16:34.552815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.552016    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" event={"ID":"f303a365-4e50-4f59-84fe-f6e83ca4c765","Type":"ContainerDied","Data":"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"}
Nov 01 22:16:34.552815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.552067    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" event={"ID":"f303a365-4e50-4f59-84fe-f6e83ca4c765","Type":"ContainerDied","Data":"61c96e460bd2df347af7bc2b5c38e366e64efc7745bdf7cd82a8266c63bc5135"}
Nov 01 22:16:34.552815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.552082    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"
Nov 01 22:16:34.554676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.554650    5845 generic.go:356] "Generic (PLEG): container finished" podID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerID="8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f" exitCode=0
Nov 01 22:16:34.554945 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.554924    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"
Nov 01 22:16:34.555118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.555096    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" event={"ID":"ab0c0a23-451c-43e2-a709-d123d0dd63fb","Type":"ContainerDied","Data":"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"}
Nov 01 22:16:34.555217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.555137    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m" event={"ID":"ab0c0a23-451c-43e2-a709-d123d0dd63fb","Type":"ContainerDied","Data":"f4be840e8828251d547b6cae9709294d86ab374f412f7c7d38f1932aa45dfed5"}
Nov 01 22:16:34.577900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.577874    5845 scope.go:122] "RemoveContainer" containerID="7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"
Nov 01 22:16:34.609226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.609200    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:16:34.613016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.612991    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-qrmdd"]
Nov 01 22:16:34.622564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.621589    5845 scope.go:122] "RemoveContainer" containerID="7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"
Nov 01 22:16:34.625243 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:34.625063    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236\": container with ID starting with 7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236 not found: ID does not exist" containerID="7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"
Nov 01 22:16:34.625243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.625105    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236"} err="failed to get container status \"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236\": rpc error: code = NotFound desc = could not find container \"7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236\": container with ID starting with 7b0772d6a7e31ea528b2652dbb5ba2e83572322c89bb58324ede0c053fa24236 not found: ID does not exist"
Nov 01 22:16:34.625243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.625129    5845 scope.go:122] "RemoveContainer" containerID="8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"
Nov 01 22:16:34.630673 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.630647    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:16:34.636896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.636867    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9d8b5d-openshift-default-6f9cdb5bcc-7wf2m"]
Nov 01 22:16:34.642855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.642812    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:16:34.646781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.646760    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp"]
Nov 01 22:16:34.652209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.651901    5845 scope.go:122] "RemoveContainer" containerID="8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"
Nov 01 22:16:34.652397 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:34.652376    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f\": container with ID starting with 8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f not found: ID does not exist" containerID="8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"
Nov 01 22:16:34.652468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.652404    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f"} err="failed to get container status \"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f\": rpc error: code = NotFound desc = could not find container \"8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f\": container with ID starting with 8a8dbab7a4be10b7e67a80dcfcf1a195ec297cd68523ff4079de36ad4dcf8a5f not found: ID does not exist"
Nov 01 22:16:34.710470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.710438    5845 patch_prober.go:29] interesting pod/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:16:34.710618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.710497    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-17957f-openshift-default-64b4cbb957-mz8qp" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:16:34.819731 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:34.819646    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:16:34.863638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.863601    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" path="/var/lib/kubelet/pods/99efaa10-3ca4-4a9d-98f0-d776ffb1a467/volumes"
Nov 01 22:16:34.864132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.864117    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" path="/var/lib/kubelet/pods/ab0c0a23-451c-43e2-a709-d123d0dd63fb/volumes"
Nov 01 22:16:34.864671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:34.864606    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" path="/var/lib/kubelet/pods/f303a365-4e50-4f59-84fe-f6e83ca4c765/volumes"
Nov 01 22:16:35.114258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.114180    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:16:35.114997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.114978    5845 cpu_manager.go:414] "containerMap: removing container" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:35.114997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.114997    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33be8278-82a2-49cd-abc9-43baf840e5ea" containerName="cleanup"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115013    5845 cpu_manager.go:414] "containerMap: removing container" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115018    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115045    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115050    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115060    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115065    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115072    5845 cpu_manager.go:414] "containerMap: removing container" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115077    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eb1c81-a220-4b85-80a0-1e15e6a57b02" containerName="cleanup"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115093    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115097    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115117    5845 cpu_manager.go:414] "containerMap: removing container" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115123    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115133    5845 cpu_manager.go:414] "containerMap: removing container" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing"
Nov 01 22:16:35.115158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115138    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115239    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9b1317-41d8-46d9-8989-ff01da63c54f" containerName="cleanup"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115248    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0c0a23-451c-43e2-a709-d123d0dd63fb" containerName="istio-proxy"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115312    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f303a365-4e50-4f59-84fe-f6e83ca4c765" containerName="istio-proxy"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115331    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="37272cfa-7de1-4bdf-bd98-2ded556f162a" containerName="payload-pre-processing"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115343    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="339a5d91-271c-4509-b485-908329cdc38c" containerName="payload-processing"
Nov 01 22:16:35.115728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.115359    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="99efaa10-3ca4-4a9d-98f0-d776ffb1a467" containerName="istio-proxy"
Nov 01 22:16:35.119496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.119479    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.124084 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.124049    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-e3890e\""
Nov 01 22:16:35.124215 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.124096    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-e3890e-dockercfg-v2tzc\""
Nov 01 22:16:35.128997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.128968    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:16:35.143255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.143225    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.143466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.143400    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnhw\" (UniqueName: \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.151263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.151233    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:16:35.162087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.162058    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:16:35.162218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.162175    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.258516    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnhw\" (UniqueName: \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.258764    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstmc\" (UniqueName: \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.258806    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.258892    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.266987    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.273319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.272931    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnhw\" (UniqueName: \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\") pod \"payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.372399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.360673    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstmc\" (UniqueName: \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.372399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.360790    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.372399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.362027    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.372399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.368811    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:16:35.385375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.375558    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstmc\" (UniqueName: \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\") pod \"payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.440319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.438989    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:35.492434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.492402    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:35.571463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.571425    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerStarted","Data":"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"}
Nov 01 22:16:35.763115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.763087    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:16:35.763538 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:16:35.763514    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a76f1f_91d6_4c58_b228_bb7d44a660a3.slice/crio-88d78313d1a74defa596d9f07eb490e4dc713a710752b9601cce21bb5b4769a1 WatchSource:0}: Error finding container 88d78313d1a74defa596d9f07eb490e4dc713a710752b9601cce21bb5b4769a1: Status 404 returned error can't find the container with id 88d78313d1a74defa596d9f07eb490e4dc713a710752b9601cce21bb5b4769a1
Nov 01 22:16:35.802975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.802948    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:16:35.803616 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:16:35.803590    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d7cba2_325a_48db_99da_27c4d5b27e90.slice/crio-fea66a63fc1cb1288c393ad23213c8892ad95664f2156afc5c77a16ab00090f7 WatchSource:0}: Error finding container fea66a63fc1cb1288c393ad23213c8892ad95664f2156afc5c77a16ab00090f7: Status 404 returned error can't find the container with id fea66a63fc1cb1288c393ad23213c8892ad95664f2156afc5c77a16ab00090f7
Nov 01 22:16:35.828194 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.828169    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:16:35.829469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.829446    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:35.829606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:35.829490    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:36.596707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.596666    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" event={"ID":"19d7cba2-325a-48db-99da-27c4d5b27e90","Type":"ContainerStarted","Data":"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"}
Nov 01 22:16:36.597122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.596732    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" event={"ID":"19d7cba2-325a-48db-99da-27c4d5b27e90","Type":"ContainerStarted","Data":"fea66a63fc1cb1288c393ad23213c8892ad95664f2156afc5c77a16ab00090f7"}
Nov 01 22:16:36.597782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.597762    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:36.602417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.602389    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" event={"ID":"b5a76f1f-91d6-4c58-b228-bb7d44a660a3","Type":"ContainerStarted","Data":"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"}
Nov 01 22:16:36.602568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.602495    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:36.602568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.602521    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" event={"ID":"b5a76f1f-91d6-4c58-b228-bb7d44a660a3","Type":"ContainerStarted","Data":"88d78313d1a74defa596d9f07eb490e4dc713a710752b9601cce21bb5b4769a1"}
Nov 01 22:16:36.618410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.618368    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podStartSLOduration=1.61835598 podStartE2EDuration="1.61835598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:36.615508104 +0000 UTC m=+1899.871687878" watchObservedRunningTime="2026-11-01 22:16:36.61835598 +0000 UTC m=+1899.874535743"
Nov 01 22:16:36.636598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.636558    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podStartSLOduration=1.636545736 podStartE2EDuration="1.636545736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:36.633453539 +0000 UTC m=+1899.889633302" watchObservedRunningTime="2026-11-01 22:16:36.636545736 +0000 UTC m=+1899.892725498"
Nov 01 22:16:36.829608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.829568    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:36.829770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:36.829623    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:37.828459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:37.828423    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:37.828947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:37.828476    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:38.834337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:38.833520    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:38.834337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:38.833583    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:39.482078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:39.474691    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:16:39.485226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:39.485123    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:16:39.832372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:39.830669    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:39.832372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:39.830727    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:40.828181 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:40.828145    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:40.828712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:40.828193    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:41.828523 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:41.828482    5845 patch_prober.go:29] interesting pod/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 01 22:16:41.829037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:41.828551    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 01 22:16:42.834331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:42.834275    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:16:42.834777 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:42.834762    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:16:42.835515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:42.835496    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:16:44.475748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:44.475673    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing" containerID="cri-o://30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957" gracePeriod=25
Nov 01 22:16:44.486228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:44.486192    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing" containerID="cri-o://7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f" gracePeriod=25
Nov 01 22:16:45.409509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:45.409474    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:46.416396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:46.416358    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:47.613409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:47.613376    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:16:48.622780 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:48.622749    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:16:51.646772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.646732    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:16:51.651935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.651910    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.656893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.656869    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e3890e-serving-cert\""
Nov 01 22:16:51.666951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.666920    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:16:51.826871 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.826826    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zntb\" (UniqueName: \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.827234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.827202    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.827446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.827421    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.827523 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.827474    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.929000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.928922    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.929000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.928971    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.929233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.929066    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zntb\" (UniqueName: \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.929290 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.929258    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.929742 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:51.929455    5845 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-serving-cert: secret "maas-api-e2e-shared-a-w1-e3890e-serving-cert" not found
Nov 01 22:16:51.929742 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:51.929532    5845 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls podName:3c22bc83-8cab-44be-bead-1115064df300 nodeName:}" failed. No retries permitted until 2026-11-01 22:16:52.429512236 +0000 UTC m=+1915.685691990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls") pod "maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" (UID: "3c22bc83-8cab-44be-bead-1115064df300") : secret "maas-api-e2e-shared-a-w1-e3890e-serving-cert" not found
Nov 01 22:16:51.929742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.929732    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.931571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.931551    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:51.943908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:51.943888    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zntb\" (UniqueName: \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:52.355741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.355681    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:16:52.356204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.356171    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy" containerID="cri-o://e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3" gracePeriod=30
Nov 01 22:16:52.364417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.364378    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:16:52.364663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.364637    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy" containerID="cri-o://37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7" gracePeriod=30
Nov 01 22:16:52.369589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.369005    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:16:52.369589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.369241    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" containerID="cri-o://08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb" gracePeriod=30
Nov 01 22:16:52.465326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.464336    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:52.513317 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.501197    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:52.597320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.590605    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:52.927629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.927590    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:16:52.930622 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:16:52.930596    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c22bc83_8cab_44be_bead_1115064df300.slice/crio-b7380f1a5100858aaedb8cc07d91d293385dd4d2a6f716563c2f288ccbdad974 WatchSource:0}: Error finding container b7380f1a5100858aaedb8cc07d91d293385dd4d2a6f716563c2f288ccbdad974: Status 404 returned error can't find the container with id b7380f1a5100858aaedb8cc07d91d293385dd4d2a6f716563c2f288ccbdad974
Nov 01 22:16:52.932714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:52.932689    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:16:53.766938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:53.766853    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" event={"ID":"3c22bc83-8cab-44be-bead-1115064df300","Type":"ContainerStarted","Data":"eb329e2225445ae6c8d436eed8d15f502da9a2be9b952a837f11a83935dcc4bd"}
Nov 01 22:16:53.766938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:53.766914    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" event={"ID":"3c22bc83-8cab-44be-bead-1115064df300","Type":"ContainerStarted","Data":"b7380f1a5100858aaedb8cc07d91d293385dd4d2a6f716563c2f288ccbdad974"}
Nov 01 22:16:53.767159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:53.766945    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:16:53.790703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:53.790651    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" podStartSLOduration=2.245991694 podStartE2EDuration="2.790637412s" totalImagesPullingTime="544.645718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:16:53.788074405 +0000 UTC m=+1917.044254197" watchObservedRunningTime="2026-11-01 22:16:53.790637412 +0000 UTC m=+1917.046817174"
Nov 01 22:16:55.409052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:55.409016    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:56.416745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:56.416710    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:16:56.679909 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:56.679810    5845 patch_prober.go:29] interesting pod/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:16:56.679909 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:56.679878    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:16:57.729671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.729637    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:16:57.733468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.733447    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:16:57.776543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.776509    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:16:57.809719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.809683    5845 generic.go:356] "Generic (PLEG): container finished" podID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerID="08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb" exitCode=0
Nov 01 22:16:57.809874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.809742    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"
Nov 01 22:16:57.809874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.809762    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" event={"ID":"8a635544-6e59-4a28-b4d7-c3c74b5a300c","Type":"ContainerDied","Data":"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"}
Nov 01 22:16:57.809874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.809852    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f" event={"ID":"8a635544-6e59-4a28-b4d7-c3c74b5a300c","Type":"ContainerDied","Data":"70725dd0d6d6036ef495f6b83fef417d08c24bcf3d85d5a675f012c23e884d3a"}
Nov 01 22:16:57.810044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.809882    5845 scope.go:122] "RemoveContainer" containerID="08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"
Nov 01 22:16:57.812273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.812251    5845 generic.go:356] "Generic (PLEG): container finished" podID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerID="37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7" exitCode=0
Nov 01 22:16:57.812456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.812440    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"
Nov 01 22:16:57.812609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.812592    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" event={"ID":"7b324fc4-13b6-4e6f-b606-5f90561a3ae6","Type":"ContainerDied","Data":"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"}
Nov 01 22:16:57.812695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.812634    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s" event={"ID":"7b324fc4-13b6-4e6f-b606-5f90561a3ae6","Type":"ContainerDied","Data":"42706c6710ee6c6bf9a592ce823c91cd0ee381e60266aefa87f1b6ab6de610fc"}
Nov 01 22:16:57.815032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.815006    5845 generic.go:356] "Generic (PLEG): container finished" podID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerID="e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3" exitCode=0
Nov 01 22:16:57.815168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.815075    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" event={"ID":"a62e04ab-98b5-4a19-8430-f5cb71e677ff","Type":"ContainerDied","Data":"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"}
Nov 01 22:16:57.815168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.815121    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb" event={"ID":"a62e04ab-98b5-4a19-8430-f5cb71e677ff","Type":"ContainerDied","Data":"22773f99efd4cc2cf34ef139f9add9eafa0437df92524942c32676cdf9bda41c"}
Nov 01 22:16:57.815168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.815133    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"
Nov 01 22:16:57.838797 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.838770    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.838920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.838853    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.838920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.838886    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.838996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.838942    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.838995    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839048    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839108    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839153    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839179    5845 scope.go:122] "RemoveContainer" containerID="08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"
Nov 01 22:16:57.839262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839184    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839346 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839317    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839402    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839470    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839508    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839580    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\") pod \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\" (UID: \"a62e04ab-98b5-4a19-8430-f5cb71e677ff\") "
Nov 01 22:16:57.839671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839647    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839770    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839875    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.839965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.839951    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\") pod \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\" (UID: \"7b324fc4-13b6-4e6f-b606-5f90561a3ae6\") "
Nov 01 22:16:57.841175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.841148    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.841454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.841428    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.842134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.842112    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.842621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.842600    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.842964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.842944    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.843760 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:57.843733    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb\": container with ID starting with 08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb not found: ID does not exist" containerID="08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"
Nov 01 22:16:57.843912 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.843887    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb"} err="failed to get container status \"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb\": rpc error: code = NotFound desc = could not find container \"08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb\": container with ID starting with 08332940eaa7adfd16bbd6e5554568f0001dd92bff951356a3dded0193375cbb not found: ID does not exist"
Nov 01 22:16:57.844078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.844064    5845 scope.go:122] "RemoveContainer" containerID="37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"
Nov 01 22:16:57.844227 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.844009    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.844350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.844215    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.846154 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.846132    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:57.848625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.848211    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "kube-api-access-tdk27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:57.848625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.848350    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.848774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.848689    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:57.849371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.849228    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.849866 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.849837    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.849990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.849968    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "kube-api-access-xhgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:57.850695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.850673    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:57.851122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.851099    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token" pod "a62e04ab-98b5-4a19-8430-f5cb71e677ff" (UID: "a62e04ab-98b5-4a19-8430-f5cb71e677ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:57.852445 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.852422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:57.852620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.852600    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo" pod "7b324fc4-13b6-4e6f-b606-5f90561a3ae6" (UID: "7b324fc4-13b6-4e6f-b606-5f90561a3ae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:57.862925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.862903    5845 scope.go:122] "RemoveContainer" containerID="37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"
Nov 01 22:16:57.863237 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:57.863213    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7\": container with ID starting with 37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7 not found: ID does not exist" containerID="37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"
Nov 01 22:16:57.863379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.863244    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7"} err="failed to get container status \"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7\": rpc error: code = NotFound desc = could not find container \"37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7\": container with ID starting with 37c479500a818b6daf01b59a91c4c066460c57b01a07b3086bd4386f4059dfb7 not found: ID does not exist"
Nov 01 22:16:57.863379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.863259    5845 scope.go:122] "RemoveContainer" containerID="e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"
Nov 01 22:16:57.911892 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.910614    5845 scope.go:122] "RemoveContainer" containerID="e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"
Nov 01 22:16:57.911892 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:16:57.911099    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3\": container with ID starting with e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3 not found: ID does not exist" containerID="e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"
Nov 01 22:16:57.911892 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.911135    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3"} err="failed to get container status \"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3\": rpc error: code = NotFound desc = could not find container \"e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3\": container with ID starting with e75433c60387d111023c6fe1ced0140aa20c43216586b0993843f2e8ca37ffc3 not found: ID does not exist"
Nov 01 22:16:57.942459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942417    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942491    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942622    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942669    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942755    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942865    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.942991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.942974    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.943083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.943067    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.943150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.943125    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\") pod \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\" (UID: \"8a635544-6e59-4a28-b4d7-c3c74b5a300c\") "
Nov 01 22:16:57.944171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944144    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944177    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944195    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944209    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944223    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944239    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944252    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944268    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdk27\" (UniqueName: \"kubernetes.io/projected/a62e04ab-98b5-4a19-8430-f5cb71e677ff-kube-api-access-tdk27\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944283    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944316    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944330    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944342    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944355    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944369    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944384    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgk5\" (UniqueName: \"kubernetes.io/projected/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-kube-api-access-xhgk5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944397    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944410    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b324fc4-13b6-4e6f-b606-5f90561a3ae6-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.944678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.944423    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a62e04ab-98b5-4a19-8430-f5cb71e677ff-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:57.945240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.945155    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.945345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.945273    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.945735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.945708    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.945947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.945918    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:16:57.946280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.946257    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.948001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.947974    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "kube-api-access-kd2rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:57.949189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.949167    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:16:57.955670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.955636    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:16:57.958747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:57.958723    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token" pod "8a635544-6e59-4a28-b4d7-c3c74b5a300c" (UID: "8a635544-6e59-4a28-b4d7-c3c74b5a300c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:16:58.045718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045682    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.045718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045712    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.045718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045721    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045730    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2rp\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-kube-api-access-kd2rp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045742    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045750    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045758    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045767    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.046014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.045774    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a635544-6e59-4a28-b4d7-c3c74b5a300c-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:16:58.140284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.140258    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:16:58.144208 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.144183    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-6tf6f"]
Nov 01 22:16:58.155520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.155494    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:16:58.165504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.165440    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-9lksb"]
Nov 01 22:16:58.173011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.172960    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:16:58.177451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.177429    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-830149-openshift-default-64cc8cfc59-vxr5s"]
Nov 01 22:16:58.864235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.864202    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" path="/var/lib/kubelet/pods/7b324fc4-13b6-4e6f-b606-5f90561a3ae6/volumes"
Nov 01 22:16:58.864922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.864898    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" path="/var/lib/kubelet/pods/8a635544-6e59-4a28-b4d7-c3c74b5a300c/volumes"
Nov 01 22:16:58.865458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:58.865439    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" path="/var/lib/kubelet/pods/a62e04ab-98b5-4a19-8430-f5cb71e677ff/volumes"
Nov 01 22:16:59.781692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:16:59.781665    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:17:04.825412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.824980    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:17:04.860101 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:04.859631    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:17:04.874939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.874912    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:17:04.881524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.881494    5845 generic.go:356] "Generic (PLEG): container finished" podID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerID="7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f" exitCode=0
Nov 01 22:17:04.881659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.881531    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"
Nov 01 22:17:04.881659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.881577    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" event={"ID":"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2","Type":"ContainerDied","Data":"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"}
Nov 01 22:17:04.881659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.881639    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg" event={"ID":"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2","Type":"ContainerDied","Data":"20f52139f1415c44471c19fd0890d5ad4a7dcf5d2a2e9b1ae2ae5bc074d0ad63"}
Nov 01 22:17:04.881805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.881672    5845 scope.go:122] "RemoveContainer" containerID="7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"
Nov 01 22:17:04.884059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.884038    5845 generic.go:356] "Generic (PLEG): container finished" podID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerID="30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957" exitCode=0
Nov 01 22:17:04.884167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.884091    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"
Nov 01 22:17:04.884167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.884102    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" event={"ID":"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80","Type":"ContainerDied","Data":"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"}
Nov 01 22:17:04.884167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.884135    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m" event={"ID":"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80","Type":"ContainerDied","Data":"ba2e689bdc7e62bd7cf58782a43c385efbc6588122421e28d1637e7a2a362e98"}
Nov 01 22:17:04.904272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.904251    5845 scope.go:122] "RemoveContainer" containerID="7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"
Nov 01 22:17:04.905356 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:04.905323    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f\": container with ID starting with 7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f not found: ID does not exist" containerID="7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"
Nov 01 22:17:04.905501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.905368    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f"} err="failed to get container status \"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f\": rpc error: code = NotFound desc = could not find container \"7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f\": container with ID starting with 7b83e97b9e38410ca757742be0eda319e71b97ee2eb589fc17b8300e9149155f not found: ID does not exist"
Nov 01 22:17:04.905501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.905390    5845 scope.go:122] "RemoveContainer" containerID="30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"
Nov 01 22:17:04.916015 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.915990    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\" (UniqueName: \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\") pod \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") "
Nov 01 22:17:04.916292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.916272    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\") pod \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\" (UID: \"fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80\") "
Nov 01 22:17:04.916712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.916686    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume" pod "fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" (UID: "fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:04.917582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.917565    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:04.918602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.918579    5845 scope.go:122] "RemoveContainer" containerID="30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"
Nov 01 22:17:04.918905 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:04.918876    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957\": container with ID starting with 30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957 not found: ID does not exist" containerID="30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"
Nov 01 22:17:04.918989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.918914    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957"} err="failed to get container status \"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957\": rpc error: code = NotFound desc = could not find container \"30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957\": container with ID starting with 30bbcad6122cf02334f15709fbf37713b779a5eba6ceeb38a8d0d1ba0a773957 not found: ID does not exist"
Nov 01 22:17:04.921034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:04.921010    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg" pod "fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" (UID: "fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80"). InnerVolumeSpecName "kube-api-access-947jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:05.019291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.019216    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\" (UniqueName: \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\") pod \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") "
Nov 01 22:17:05.019466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.019411    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\") pod \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\" (UID: \"5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2\") "
Nov 01 22:17:05.019776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.019745    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume" pod "5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" (UID: "5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:05.020016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.020002    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947jg\" (UniqueName: \"kubernetes.io/projected/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80-kube-api-access-947jg\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:05.020067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.020020    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:05.021342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.021285    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn" pod "5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" (UID: "5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2"). InnerVolumeSpecName "kube-api-access-2jtgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:05.121801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.121773    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jtgn\" (UniqueName: \"kubernetes.io/projected/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2-kube-api-access-2jtgn\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:05.219077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.219044    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:17:05.222387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.222359    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-830149-58d9646cff-lnzrg"]
Nov 01 22:17:05.226713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.226687    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:17:05.229867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:05.229846    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-830149-7f864c9c64-rvq4m"]
Nov 01 22:17:06.205116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:06.205085    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:17:06.862650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:06.862610    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" path="/var/lib/kubelet/pods/5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2/volumes"
Nov 01 22:17:06.863147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:06.863129    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" path="/var/lib/kubelet/pods/fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80/volumes"
Nov 01 22:17:11.205683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:11.205635    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing" containerID="cri-o://177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15" gracePeriod=25
Nov 01 22:17:14.335328 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:14.335251    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:17:15.253935 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.253896    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:17:15.255116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255094    5845 cpu_manager.go:414] "containerMap: removing container" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing"
Nov 01 22:17:15.255116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255115    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255145    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255151    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255165    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255170    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255206    5845 cpu_manager.go:414] "containerMap: removing container" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255211    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255221    5845 cpu_manager.go:414] "containerMap: removing container" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy"
Nov 01 22:17:15.255278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255226    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy"
Nov 01 22:17:15.255570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255355    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3a7de6-ee97-44e4-b5ee-2dbc1ac2eba2" containerName="payload-pre-processing"
Nov 01 22:17:15.255570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255367    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62e04ab-98b5-4a19-8430-f5cb71e677ff" containerName="istio-proxy"
Nov 01 22:17:15.255570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255376    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b324fc4-13b6-4e6f-b606-5f90561a3ae6" containerName="istio-proxy"
Nov 01 22:17:15.255570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255388    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7ce88d-7eac-476e-9a1c-1fad0d0b8e80" containerName="payload-processing"
Nov 01 22:17:15.255570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.255395    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a635544-6e59-4a28-b4d7-c3c74b5a300c" containerName="istio-proxy"
Nov 01 22:17:15.260920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.260901    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.265515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.265465    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-51e3b7-openshift-default-dockercfg-5ns2r\""
Nov 01 22:17:15.269226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.269203    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.401763    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml9k5\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.401893    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.401924    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.402073    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.402366    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.402518    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.407817    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.407853    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.434339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.407943    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.520621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.520527    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml9k5\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.520948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.520927    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.521528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.521502    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.521868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.521850    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.522416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.522394    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.522687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.522667    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.522888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.522873    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.522755    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.522448    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.523052    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.537939    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.521445    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.539274    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.539867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.539832    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.540315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.540271    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.549801 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.547820    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.561962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.558061    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml9k5\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.570324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.569368    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\") pod \"e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") " pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.585678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.584408    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:15.916553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.916522    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:17:15.917740 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:17:15.917713    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36470550_abf4_47f8_8c5e_583fc7e875d5.slice/crio-1001036b2111b505314c601d94e0d539acc76d585c53adc8c5d52eccdc632743 WatchSource:0}: Error finding container 1001036b2111b505314c601d94e0d539acc76d585c53adc8c5d52eccdc632743: Status 404 returned error can't find the container with id 1001036b2111b505314c601d94e0d539acc76d585c53adc8c5d52eccdc632743
Nov 01 22:17:15.920600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.920573    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:15.920681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.920633    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:15.920681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:15.920663    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:16.036400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:16.036364    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerStarted","Data":"1001036b2111b505314c601d94e0d539acc76d585c53adc8c5d52eccdc632743"}
Nov 01 22:17:16.061859 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:16.061797    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podStartSLOduration=1.061783111 podStartE2EDuration="1.061783111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:16.058535844 +0000 UTC m=+1939.314715610" watchObservedRunningTime="2026-11-01 22:17:16.061783111 +0000 UTC m=+1939.317962874"
Nov 01 22:17:16.584808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:16.584780    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:17.086543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:17.084364    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerStarted","Data":"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"}
Nov 01 22:17:17.600023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:17.599984    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:18.095494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:18.095465    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:18.096750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:18.096722    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:17:19.701788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.701753    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:17:19.706770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.706749    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.712000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.711945    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-51e3b7-dockercfg-2nlk2\""
Nov 01 22:17:19.715466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.715444    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:17:19.784650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.784627    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-51e3b7\""
Nov 01 22:17:19.838352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.831644    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n89v\" (UniqueName: \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.838352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.832032    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.935325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.933639    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.935325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.933772    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n89v\" (UniqueName: \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.935325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.935155    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:19.961369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:19.954469    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n89v\" (UniqueName: \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\") pod \"payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:20.027661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:20.027608    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:20.348042 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:20.348013    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:17:20.348897 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:17:20.348872    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf123a7c8_5b58_4630_8eb1_4984e30e9cfb.slice/crio-ab151241ac2f52928c6cd224d838867bac7bc53b1e3bd533f4d447c0cf8a0ab2 WatchSource:0}: Error finding container ab151241ac2f52928c6cd224d838867bac7bc53b1e3bd533f4d447c0cf8a0ab2: Status 404 returned error can't find the container with id ab151241ac2f52928c6cd224d838867bac7bc53b1e3bd533f4d447c0cf8a0ab2
Nov 01 22:17:21.123745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:21.123703    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" event={"ID":"f123a7c8-5b58-4630-8eb1-4984e30e9cfb","Type":"ContainerStarted","Data":"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"}
Nov 01 22:17:21.124201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:21.123776    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" event={"ID":"f123a7c8-5b58-4630-8eb1-4984e30e9cfb","Type":"ContainerStarted","Data":"ab151241ac2f52928c6cd224d838867bac7bc53b1e3bd533f4d447c0cf8a0ab2"}
Nov 01 22:17:21.124201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:21.123815    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:21.145617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:21.145568    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podStartSLOduration=2.145551196 podStartE2EDuration="2.145551196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:21.141613037 +0000 UTC m=+1944.397792803" watchObservedRunningTime="2026-11-01 22:17:21.145551196 +0000 UTC m=+1944.401730960"
Nov 01 22:17:22.440786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:22.440749    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:17:22.441727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:22.441675    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy" containerID="cri-o://b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2" gracePeriod=30
Nov 01 22:17:24.330417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:24.330380    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:17:27.940164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:27.940131    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:17:28.019765 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.019714    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.019765 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.019764    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.019814    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.019880    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.019912    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.020010    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.020124    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.020353    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.020385    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\") pod \"e45395db-02f5-4997-ba4a-82626497f3d0\" (UID: \"e45395db-02f5-4997-ba4a-82626497f3d0\") "
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.021493    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:28.022641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.021976    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:28.023147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.023016    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:28.023487 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.023463    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:28.023693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.023675    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:28.136520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.132688    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45395db-02f5-4997-ba4a-82626497f3d0-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.136520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.132729    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.136520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.132745    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.136520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.132760    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.136520 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.132775    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.142370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.137284    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "kube-api-access-8svlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:28.201380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.197017    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:28.207326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.206500    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:28.207326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.206759    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo" pod "e45395db-02f5-4997-ba4a-82626497f3d0" (UID: "e45395db-02f5-4997-ba4a-82626497f3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:17:28.234011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.233939    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45395db-02f5-4997-ba4a-82626497f3d0-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.234011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.233969    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45395db-02f5-4997-ba4a-82626497f3d0-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.234011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.233979    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8svlw\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-kube-api-access-8svlw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.234011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.233990    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45395db-02f5-4997-ba4a-82626497f3d0-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:28.343929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.343886    5845 generic.go:356] "Generic (PLEG): container finished" podID="e45395db-02f5-4997-ba4a-82626497f3d0" containerID="b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2" exitCode=0
Nov 01 22:17:28.344173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.344140    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" event={"ID":"e45395db-02f5-4997-ba4a-82626497f3d0","Type":"ContainerDied","Data":"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"}
Nov 01 22:17:28.344263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.344197    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd" event={"ID":"e45395db-02f5-4997-ba4a-82626497f3d0","Type":"ContainerDied","Data":"2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a"}
Nov 01 22:17:28.344263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.344229    5845 scope.go:122] "RemoveContainer" containerID="b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"
Nov 01 22:17:28.344470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.344445    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"
Nov 01 22:17:28.509666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.509632    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:17:28.516590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.516508    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-487639-openshift-default-5d5fc9449c-hcfbd"]
Nov 01 22:17:28.523693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.523342    5845 scope.go:122] "RemoveContainer" containerID="b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"
Nov 01 22:17:28.524506 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:28.524421    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2\": container with ID starting with b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2 not found: ID does not exist" containerID="b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"
Nov 01 22:17:28.524506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.524462    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2"} err="failed to get container status \"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2\": rpc error: code = NotFound desc = could not find container \"b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2\": container with ID starting with b163d0f401006a4a84c41228feb6f4c1eaef6f5473b3a00ef9af22bbf580b8f2 not found: ID does not exist"
Nov 01 22:17:28.632668 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:28.632633    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45395db_02f5_4997_ba4a_82626497f3d0.slice/crio-2d33531438e3cdf8b557e4dac0199514d7e487ec6afdb6d3d8909d4c6f8b386a\": RecentStats: unable to find data in memory cache]"
Nov 01 22:17:28.891342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:28.886281    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" path="/var/lib/kubelet/pods/e45395db-02f5-4997-ba4a-82626497f3d0/volumes"
Nov 01 22:17:31.846100 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.846076    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:17:31.927646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.927607    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\" (UniqueName: \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\") pod \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") "
Nov 01 22:17:31.927834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.927676    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\") pod \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\" (UID: \"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2\") "
Nov 01 22:17:31.928076 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.928049    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume" pod "5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" (UID: "5c119c05-2a1c-4e9a-ba8f-f6b9769458c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:31.929017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.928989    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:31.972884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:31.972845    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7" pod "5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" (UID: "5c119c05-2a1c-4e9a-ba8f-f6b9769458c2"). InnerVolumeSpecName "kube-api-access-r4vj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:32.053321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.043262    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4vj7\" (UniqueName: \"kubernetes.io/projected/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2-kube-api-access-r4vj7\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:32.139509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.139478    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:17:32.408921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.408834    5845 generic.go:356] "Generic (PLEG): container finished" podID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerID="177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15" exitCode=0
Nov 01 22:17:32.408921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.408882    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"
Nov 01 22:17:32.409101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.408921    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" event={"ID":"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2","Type":"ContainerDied","Data":"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"}
Nov 01 22:17:32.409101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.408978    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8" event={"ID":"5c119c05-2a1c-4e9a-ba8f-f6b9769458c2","Type":"ContainerDied","Data":"1fa13a71ff068f7d88e00d02447335feae31f9afb50b22d995be6607e905b6e9"}
Nov 01 22:17:32.409101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.409000    5845 scope.go:122] "RemoveContainer" containerID="177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"
Nov 01 22:17:32.422751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.422733    5845 scope.go:122] "RemoveContainer" containerID="177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"
Nov 01 22:17:32.423027 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:32.423004    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15\": container with ID starting with 177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15 not found: ID does not exist" containerID="177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"
Nov 01 22:17:32.423101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.423039    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15"} err="failed to get container status \"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15\": rpc error: code = NotFound desc = could not find container \"177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15\": container with ID starting with 177b43066ae3ab64f5ed2a4daab31cf4a2065a1234a161fecef9ce1bac0b6e15 not found: ID does not exist"
Nov 01 22:17:32.433946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.433913    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:17:32.439242 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.439220    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-487639-cdbf7c87d-7txk8"]
Nov 01 22:17:32.867469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:32.867426    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" path="/var/lib/kubelet/pods/5c119c05-2a1c-4e9a-ba8f-f6b9769458c2/volumes"
Nov 01 22:17:33.520788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:33.520756    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:17:33.566541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:33.565465    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:17:33.566541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:33.565522    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:35.222463 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:17:35.222423    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:17:38.521549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:38.521476    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" podUID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" containerName="payload-processing" containerID="cri-o://a70f67a2b087dae310823301707f9041d15fd6dfd76b0d80b1db5934e38c4187" gracePeriod=25
Nov 01 22:17:39.154002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:39.153964    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:17:39.154275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:39.154244    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" podUID="80b81845-ae0d-4275-a003-b475056ff7fe" containerName="maas-api" containerID="cri-o://49ca1d1886fd2df008b82dd9579fa8ab9c71578d8fa7d82017371521d9bf8d89" gracePeriod=30
Nov 01 22:17:39.521327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:39.517348    5845 generic.go:356] "Generic (PLEG): container finished" podID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" containerID="a70f67a2b087dae310823301707f9041d15fd6dfd76b0d80b1db5934e38c4187" exitCode=0
Nov 01 22:17:39.521327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:39.517548    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" event={"ID":"ccf3023a-050b-4a7b-a93b-047a3328e1e9","Type":"ContainerDied","Data":"a70f67a2b087dae310823301707f9041d15fd6dfd76b0d80b1db5934e38c4187"}
Nov 01 22:17:40.524560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.524479    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:17:40.527426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.527396    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"
Nov 01 22:17:40.527568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.527413    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh" event={"ID":"ccf3023a-050b-4a7b-a93b-047a3328e1e9","Type":"ContainerDied","Data":"957533202b7fd32025f8bf69f5538036bda5206d8e901f7bddb88990875ac0ab"}
Nov 01 22:17:40.527568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.527496    5845 scope.go:122] "RemoveContainer" containerID="a70f67a2b087dae310823301707f9041d15fd6dfd76b0d80b1db5934e38c4187"
Nov 01 22:17:40.529393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.529373    5845 generic.go:356] "Generic (PLEG): container finished" podID="80b81845-ae0d-4275-a003-b475056ff7fe" containerID="49ca1d1886fd2df008b82dd9579fa8ab9c71578d8fa7d82017371521d9bf8d89" exitCode=0
Nov 01 22:17:40.529509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.529402    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" event={"ID":"80b81845-ae0d-4275-a003-b475056ff7fe","Type":"ContainerDied","Data":"49ca1d1886fd2df008b82dd9579fa8ab9c71578d8fa7d82017371521d9bf8d89"}
Nov 01 22:17:40.529509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.529431    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv" event={"ID":"80b81845-ae0d-4275-a003-b475056ff7fe","Type":"ContainerDied","Data":"de05b176a38707bd82183e1298dd305c26de227ecf2bc497f9149a4ac2514e48"}
Nov 01 22:17:40.529509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.529450    5845 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de05b176a38707bd82183e1298dd305c26de227ecf2bc497f9149a4ac2514e48"
Nov 01 22:17:40.605128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.605095    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\" (UniqueName: \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\") pod \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") "
Nov 01 22:17:40.605292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.605246    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\") pod \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\" (UID: \"ccf3023a-050b-4a7b-a93b-047a3328e1e9\") "
Nov 01 22:17:40.605676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.605641    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume" pod "ccf3023a-050b-4a7b-a93b-047a3328e1e9" (UID: "ccf3023a-050b-4a7b-a93b-047a3328e1e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:40.606136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.606113    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccf3023a-050b-4a7b-a93b-047a3328e1e9-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:40.763899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.763856    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:17:40.800887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.800805    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh" pod "ccf3023a-050b-4a7b-a93b-047a3328e1e9" (UID: "ccf3023a-050b-4a7b-a93b-047a3328e1e9"). InnerVolumeSpecName "kube-api-access-vjbsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:40.814604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.814571    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\") pod \"80b81845-ae0d-4275-a003-b475056ff7fe\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") "
Nov 01 22:17:40.814791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.814684    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\") pod \"80b81845-ae0d-4275-a003-b475056ff7fe\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") "
Nov 01 22:17:40.814874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.814830    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\") pod \"80b81845-ae0d-4275-a003-b475056ff7fe\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") "
Nov 01 22:17:40.815392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.815053    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\" (UniqueName: \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\") pod \"80b81845-ae0d-4275-a003-b475056ff7fe\" (UID: \"80b81845-ae0d-4275-a003-b475056ff7fe\") "
Nov 01 22:17:40.816207 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.816181    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca" pod "80b81845-ae0d-4275-a003-b475056ff7fe" (UID: "80b81845-ae0d-4275-a003-b475056ff7fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:40.819957 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.819928    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbsh\" (UniqueName: \"kubernetes.io/projected/ccf3023a-050b-4a7b-a93b-047a3328e1e9-kube-api-access-vjbsh\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:40.820067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.819961    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b81845-ae0d-4275-a003-b475056ff7fe-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:40.824719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.824685    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs" pod "80b81845-ae0d-4275-a003-b475056ff7fe" (UID: "80b81845-ae0d-4275-a003-b475056ff7fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:17:40.824834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.824778    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls" pod "80b81845-ae0d-4275-a003-b475056ff7fe" (UID: "80b81845-ae0d-4275-a003-b475056ff7fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:17:40.826367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.826342    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b" pod "80b81845-ae0d-4275-a003-b475056ff7fe" (UID: "80b81845-ae0d-4275-a003-b475056ff7fe"). InnerVolumeSpecName "kube-api-access-bp85b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:40.875592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.875563    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:17:40.879352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.879320    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3e300c-74b58496db-grjgh"]
Nov 01 22:17:40.924441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.924399    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:40.924441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.924435    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp85b\" (UniqueName: \"kubernetes.io/projected/80b81845-ae0d-4275-a003-b475056ff7fe-kube-api-access-bp85b\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:40.924752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:40.924451    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b81845-ae0d-4275-a003-b475056ff7fe-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:41.537713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:41.537685    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"
Nov 01 22:17:41.577846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:41.577811    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:17:41.582571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:41.582542    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3e300c-7d9fc86b85-kv6qv"]
Nov 01 22:17:42.941959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:42.941926    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b81845-ae0d-4275-a003-b475056ff7fe" path="/var/lib/kubelet/pods/80b81845-ae0d-4275-a003-b475056ff7fe/volumes"
Nov 01 22:17:42.942469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:42.942451    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" path="/var/lib/kubelet/pods/ccf3023a-050b-4a7b-a93b-047a3328e1e9/volumes"
Nov 01 22:17:43.905470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:43.905426    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:17:43.905740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:43.905497    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:44.960168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.960133    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:17:44.961099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961076    5845 cpu_manager.go:414] "containerMap: removing container" podUID="80b81845-ae0d-4275-a003-b475056ff7fe" containerName="maas-api"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961102    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b81845-ae0d-4275-a003-b475056ff7fe" containerName="maas-api"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961149    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961158    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961199    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" containerName="payload-processing"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961206    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" containerName="payload-processing"
Nov 01 22:17:44.961224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961221    5845 cpu_manager.go:414] "containerMap: removing container" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing"
Nov 01 22:17:44.961554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961228    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing"
Nov 01 22:17:44.961554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961429    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf3023a-050b-4a7b-a93b-047a3328e1e9" containerName="payload-processing"
Nov 01 22:17:44.961554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961446    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c119c05-2a1c-4e9a-ba8f-f6b9769458c2" containerName="payload-processing"
Nov 01 22:17:44.961554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961462    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b81845-ae0d-4275-a003-b475056ff7fe" containerName="maas-api"
Nov 01 22:17:44.961554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:44.961480    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45395db-02f5-4997-ba4a-82626497f3d0" containerName="istio-proxy"
Nov 01 22:17:45.045481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.045439    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:17:45.045661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.045492    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:17:45.047120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.047088    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.067284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.067252    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:17:45.067430 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.067396    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.092864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.092830    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093011    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093121 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093103    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093219    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093261    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093282    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093434    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093477    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.093514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.093500    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.121680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.121647    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-702002-openshift-default-dockercfg-2lbqw\""
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.217031    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.217081    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.217110    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.217206    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218057    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218168    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218209    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218251    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218276    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218358    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218462    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218522    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqjv\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218581    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218657    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218727    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218762    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218803    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.218832    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.221368    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.221707    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.222098    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.223617    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.224369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.224215    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.239120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.236725    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:17:45.247179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.247145    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:17:45.247387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.247254    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.247387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.247366    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.250220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.250130    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.255811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.255786    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.257452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.257428    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\") pod \"e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") " pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.321055 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321026    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321104    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjxf\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321196    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321260    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321353    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321433 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321396    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321426    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321457    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321528    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321556    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321688    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321702    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqjv\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321765    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321782    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321834    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.321952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321851    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321889    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.321952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321929    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.322177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.321960    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.322177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.322050    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.322177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.322091    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.322465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.322445    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.322552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.322479    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.324164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.324143    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.324447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.324428    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.333579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.333555    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.333689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.333673    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqjv\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.373970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.373945    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:45.396888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.396867    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:45.423511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423485    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.423591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423531    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.423737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423720    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423768    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423794    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423831    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423911    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.423924    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.424030    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjxf\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.424064    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.424461    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.424475    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.424704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.424543    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.425195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.425128    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.425984 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.425959    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.426975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.426953    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.436003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.435980    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.436102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.436080    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjxf\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\") pod \"e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") " pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.610494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.610158    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:45.826714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.826633    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:17:45.837360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.832871    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:17:45.838671 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:17:45.838645    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9a1de9_e72f_488f_8c08_83a4de512f2e.slice/crio-4dda4f369f4799601659c6cdfe866625a62fcc0363c7d408ac575e5db5062a30 WatchSource:0}: Error finding container 4dda4f369f4799601659c6cdfe866625a62fcc0363c7d408ac575e5db5062a30: Status 404 returned error can't find the container with id 4dda4f369f4799601659c6cdfe866625a62fcc0363c7d408ac575e5db5062a30
Nov 01 22:17:45.842654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.842621    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.842764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.842704    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.842764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.842750    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.844182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.844152    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.844255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.844213    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.844255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.844238    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.973463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.973434    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:17:45.977947 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:17:45.977922    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2596176_d07d_49e5_b87f_b285e49045a6.slice/crio-76aaf90774b68b99a5ce75d2dbef7dcabeb21a4c719b5f8d90deaf32f6720030 WatchSource:0}: Error finding container 76aaf90774b68b99a5ce75d2dbef7dcabeb21a4c719b5f8d90deaf32f6720030: Status 404 returned error can't find the container with id 76aaf90774b68b99a5ce75d2dbef7dcabeb21a4c719b5f8d90deaf32f6720030
Nov 01 22:17:45.982749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.982715    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.982854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.982789    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:45.982854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:45.982822    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:46.583557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.583516    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" event={"ID":"6a678a55-3332-4a67-921b-3d4bb53750a4","Type":"ContainerStarted","Data":"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"}
Nov 01 22:17:46.583768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.583617    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" event={"ID":"6a678a55-3332-4a67-921b-3d4bb53750a4","Type":"ContainerStarted","Data":"ac3893aa58d5faf4e477050afc8bee6f58d9713673ebd180acd4e8fbc24dab13"}
Nov 01 22:17:46.585910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.585600    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" event={"ID":"cf9a1de9-e72f-488f-8c08-83a4de512f2e","Type":"ContainerStarted","Data":"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"}
Nov 01 22:17:46.585910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.585650    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" event={"ID":"cf9a1de9-e72f-488f-8c08-83a4de512f2e","Type":"ContainerStarted","Data":"4dda4f369f4799601659c6cdfe866625a62fcc0363c7d408ac575e5db5062a30"}
Nov 01 22:17:46.588366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.588147    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" event={"ID":"b2596176-d07d-49e5-b87f-b285e49045a6","Type":"ContainerStarted","Data":"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"}
Nov 01 22:17:46.588366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.588205    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" event={"ID":"b2596176-d07d-49e5-b87f-b285e49045a6","Type":"ContainerStarted","Data":"76aaf90774b68b99a5ce75d2dbef7dcabeb21a4c719b5f8d90deaf32f6720030"}
Nov 01 22:17:46.609727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.609680    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podStartSLOduration=2.609667276 podStartE2EDuration="2.609667276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.608058905 +0000 UTC m=+1969.864238666" watchObservedRunningTime="2026-11-01 22:17:46.609667276 +0000 UTC m=+1969.865847039"
Nov 01 22:17:46.611026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.610992    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:46.636188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.636140    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podStartSLOduration=2.6361254819999997 podStartE2EDuration="2.636125482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.635488498 +0000 UTC m=+1969.891668262" watchObservedRunningTime="2026-11-01 22:17:46.636125482 +0000 UTC m=+1969.892305245"
Nov 01 22:17:46.662149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:46.662106    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podStartSLOduration=1.662093126 podStartE2EDuration="1.662093126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:46.657187718 +0000 UTC m=+1969.913367483" watchObservedRunningTime="2026-11-01 22:17:46.662093126 +0000 UTC m=+1969.918272888"
Nov 01 22:17:47.374814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.374782    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:47.375950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.375929    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:47.376014 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.375974    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:47.397097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.397067    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:47.398325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.398285    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:47.398449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.398347    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:47.612881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.612839    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:47.613076 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.612891    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:47.772489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.772455    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:17:47.772723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.772697    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" containerID="cri-o://656b2b3ab08fed1bccd17a0636f91333f87f68cc3e5a33c964db4f50fe8c355b" gracePeriod=30
Nov 01 22:17:47.794277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.779618    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:17:47.794277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.779833    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy" containerID="cri-o://7e1687925f3863aa39d235fa08d56f3af48e294803cc81f90711dbcd99b57415" gracePeriod=30
Nov 01 22:17:47.804394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.804335    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:17:47.804639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:47.804606    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" containerID="cri-o://776a37f3bb38da1e6ebf37dd76643e58ec42b11fc5265fea8861593ae214c34a" gracePeriod=30
Nov 01 22:17:48.379510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.379016    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:48.379510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.379081    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:48.405592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.400620    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:48.405592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.400688    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:48.579472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.579412    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:17:48.579693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.579481    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:48.624107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.624047    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:48.624337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:48.624112    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:49.029335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.029277    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:17:49.077201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.044528    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.225088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.213793    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:17:49.225088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.213859    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:17:49.226180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.226093    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-521734-openshift-default-dockercfg-wmbjz\""
Nov 01 22:17:49.248566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.044422    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:17:49.309238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.307888    5845 patch_prober.go:29] interesting pod/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:17:49.309238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.307950    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347339    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347473    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347557    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347594    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347662    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347717    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347779    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347914    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj77\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.349378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.347950    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.393344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.383067    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:49.393344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.383125    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:49.421576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.421512    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:49.421960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.421848    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:49.503082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.481711    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj77\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.503082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.498699    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.504368    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.504592    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.504726    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.538388    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.538546    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.538639    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.538754    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.540647    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.545816    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.546052    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.560503    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.563363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.560997    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.619351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.615012    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:49.619351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.615071    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:49.640394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.624329    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.737356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.630641    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.737356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.673356    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj77\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.738713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.738475    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:49.769497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:49.765638    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:50.466466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.411405    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:50.466466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.411469    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:50.466466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.438374    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:50.466466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.438433    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:50.630367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.615385    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:50.630367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:50.615453    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:51.041154 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.040160    5845 generic.go:356] "Generic (PLEG): container finished" podID="426bc111-0f0a-4c63-a50a-519e47476539" containerID="569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5" exitCode=0
Nov 01 22:17:51.239534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.237761    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerDied","Data":"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"}
Nov 01 22:17:51.239534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.238444    5845 scope.go:122] "RemoveContainer" containerID="569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"
Nov 01 22:17:51.321550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.298199    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:51.321550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.298344    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:51.321550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.298406    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:51.396453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.396416    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:51.499410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.452658    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:51.663562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.654453    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:51.763674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.741446    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:51.763674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.743349    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:51.763674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:51.743424    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:52.042456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.042409    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:52.042772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.042740    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:17:52.139730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.137059    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:52.367173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.345127    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:52.367173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.345224    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:52.367173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.345267    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:52.439211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.438534    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:52.439211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.438593    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:52.439211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.438678    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:52.439211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.438704    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:52.640109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.633267    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:52.640109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.634469    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:52.640109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:52.637376    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:17:53.428988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.428483    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:53.428988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.428547    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:53.449321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.444456    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:53.449321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.444513    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:53.528205 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.528119    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerStarted","Data":"a87a6440b71e063bd06a461dc2868d78d938c864a8af2aac702dcd9031719190"}
Nov 01 22:17:53.530794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.530640    5845 generic.go:356] "Generic (PLEG): container finished" podID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerID="656b2b3ab08fed1bccd17a0636f91333f87f68cc3e5a33c964db4f50fe8c355b" exitCode=0
Nov 01 22:17:53.530794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.530724    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" event={"ID":"c70e5a04-ae17-4f6b-bfa7-4908166d4e33","Type":"ContainerDied","Data":"656b2b3ab08fed1bccd17a0636f91333f87f68cc3e5a33c964db4f50fe8c355b"}
Nov 01 22:17:53.533672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.533651    5845 generic.go:356] "Generic (PLEG): container finished" podID="a180a956-8b70-4fba-855b-eacd78570803" containerID="776a37f3bb38da1e6ebf37dd76643e58ec42b11fc5265fea8861593ae214c34a" exitCode=0
Nov 01 22:17:53.533933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.533871    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" event={"ID":"a180a956-8b70-4fba-855b-eacd78570803","Type":"ContainerDied","Data":"776a37f3bb38da1e6ebf37dd76643e58ec42b11fc5265fea8861593ae214c34a"}
Nov 01 22:17:53.540360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.540330    5845 generic.go:356] "Generic (PLEG): container finished" podID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerID="7e1687925f3863aa39d235fa08d56f3af48e294803cc81f90711dbcd99b57415" exitCode=0
Nov 01 22:17:53.540656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.540612    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerDied","Data":"7e1687925f3863aa39d235fa08d56f3af48e294803cc81f90711dbcd99b57415"}
Nov 01 22:17:53.540822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.540808    5845 scope.go:122] "RemoveContainer" containerID="a86a5daa0974e76fd6554636de7018179747532175030083000f29c213ecbdf3"
Nov 01 22:17:53.617897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.612286    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:17:53.617897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:53.614718    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 01 22:17:54.378126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.377723    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 01 22:17:54.378126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.377781    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 01 22:17:54.439455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.436056    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 01 22:17:54.439455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.436115    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 01 22:17:54.542426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.542392    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:17:54.638124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.634455    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:54.654489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.654407    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:54.682841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.677979    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:17:54.686485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.686393    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:17:54.724152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.724118    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm" event={"ID":"b48f8f3a-093e-43f4-84cc-cbd03315ef3f","Type":"ContainerDied","Data":"e78db7d84684a47156965a6c9673dc5ed3e1aa8ee4308b8d8254ef6527433667"}
Nov 01 22:17:54.725360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.725338    5845 scope.go:122] "RemoveContainer" containerID="7e1687925f3863aa39d235fa08d56f3af48e294803cc81f90711dbcd99b57415"
Nov 01 22:17:54.725591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.724461    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"
Nov 01 22:17:54.726160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.726144    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:17:54.727525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.727501    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerStarted","Data":"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"}
Nov 01 22:17:54.730898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.730830    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerStarted","Data":"6efd6b77be58641f83b2c999b9d033877be00225997d0bedafb510ce54e4f838"}
Nov 01 22:17:54.733017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.732791    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c" event={"ID":"a180a956-8b70-4fba-855b-eacd78570803","Type":"ContainerDied","Data":"7411cdb6de3ae8715e94942ce547e51bf45ed7a59ae699ace1b227470c1e9477"}
Nov 01 22:17:54.733017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.732877    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"
Nov 01 22:17:54.798792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.794718    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828259    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828371    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828503    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828543    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828592    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828669    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828784    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.828904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828854    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.829540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.828976    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\") pod \"a180a956-8b70-4fba-855b-eacd78570803\" (UID: \"a180a956-8b70-4fba-855b-eacd78570803\") "
Nov 01 22:17:54.839600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.839318    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "kube-api-access-dzqxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:54.841368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.840117    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.842270 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.842035    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.875851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.850929    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.875851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.852006    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.875851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.852243    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:54.875851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.853683    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.875851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.858002    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podStartSLOduration=6.8579832849999995 podStartE2EDuration="6.857983285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:17:54.85294029 +0000 UTC m=+1978.109120079" watchObservedRunningTime="2026-11-01 22:17:54.857983285 +0000 UTC m=+1978.114163048"
Nov 01 22:17:54.888214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.887172    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:54.889236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.889193    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo" pod "a180a956-8b70-4fba-855b-eacd78570803" (UID: "a180a956-8b70-4fba-855b-eacd78570803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935522    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935647    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935679    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935750    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935823    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935890    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935931    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.935987    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936056    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936134    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936204    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936380    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936452    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936493    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936551    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936603    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936665    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\") pod \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\" (UID: \"b48f8f3a-093e-43f4-84cc-cbd03315ef3f\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936721    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\") pod \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\" (UID: \"c70e5a04-ae17-4f6b-bfa7-4908166d4e33\") "
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.936743    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.937233    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.938023    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.938383    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.938768    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.938942    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.939777    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942132    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942158    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942175    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a180a956-8b70-4fba-855b-eacd78570803-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942188    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a180a956-8b70-4fba-855b-eacd78570803-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942203    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942217    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942230    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942244    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942258    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942272    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942286    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942319    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzqxh\" (UniqueName: \"kubernetes.io/projected/a180a956-8b70-4fba-855b-eacd78570803-kube-api-access-dzqxh\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942334    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942348    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a180a956-8b70-4fba-855b-eacd78570803-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942364    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942378    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.942503    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.943915    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.944121    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.945766    5845 scope.go:122] "RemoveContainer" containerID="776a37f3bb38da1e6ebf37dd76643e58ec42b11fc5265fea8861593ae214c34a"
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.950930    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.951790    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.953960    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.955691    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:17:54.974377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:54.968943    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.097952    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.097992    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098008    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098024    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098038    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098051    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098063    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.104080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.098074    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.119122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.112501    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "kube-api-access-5w649". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:55.119122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.114696    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token" pod "b48f8f3a-093e-43f4-84cc-cbd03315ef3f" (UID: "b48f8f3a-093e-43f4-84cc-cbd03315ef3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:55.119122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.116545    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl" pod "c70e5a04-ae17-4f6b-bfa7-4908166d4e33" (UID: "c70e5a04-ae17-4f6b-bfa7-4908166d4e33"). InnerVolumeSpecName "kube-api-access-2cxgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:17:55.135807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.135736    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:17:55.140411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.140384    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-8db5c"]
Nov 01 22:17:55.200264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.200155    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxgl\" (UniqueName: \"kubernetes.io/projected/c70e5a04-ae17-4f6b-bfa7-4908166d4e33-kube-api-access-2cxgl\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.200264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.200203    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w649\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-kube-api-access-5w649\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.200264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.200223    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48f8f3a-093e-43f4-84cc-cbd03315ef3f-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:17:55.376426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.369659    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:17:55.378063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.378029    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-dxhsm"]
Nov 01 22:17:55.381037 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.381013    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:55.381508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.381421    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:55.382006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.381990    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:17:55.403892 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.403869    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:55.404189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.404169    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:55.405412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.405391    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:17:55.519677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.519631    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:17:55.520629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.520594    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:17:55.520894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.520655    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:17:55.772353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.757456    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2" event={"ID":"c70e5a04-ae17-4f6b-bfa7-4908166d4e33","Type":"ContainerDied","Data":"8cbc476cb05e58354f2b39b64367ca9ea310004ad9043a6b41d091a1b3374583"}
Nov 01 22:17:55.772353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.757533    5845 scope.go:122] "RemoveContainer" containerID="656b2b3ab08fed1bccd17a0636f91333f87f68cc3e5a33c964db4f50fe8c355b"
Nov 01 22:17:55.772353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.757681    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"
Nov 01 22:17:55.801586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.801562    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:55.819357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.819162    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:17:55.826604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:55.826574    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3e300c-openshift-default-679f5f7777-c5sn2"]
Nov 01 22:17:56.520288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.520246    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:17:56.520744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.520343    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:17:56.790275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.790187    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:56.791121 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.791096    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:17:56.892096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.892061    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a180a956-8b70-4fba-855b-eacd78570803" path="/var/lib/kubelet/pods/a180a956-8b70-4fba-855b-eacd78570803/volumes"
Nov 01 22:17:56.893432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.893404    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" path="/var/lib/kubelet/pods/b48f8f3a-093e-43f4-84cc-cbd03315ef3f/volumes"
Nov 01 22:17:56.893931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:56.893916    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" path="/var/lib/kubelet/pods/c70e5a04-ae17-4f6b-bfa7-4908166d4e33/volumes"
Nov 01 22:17:57.519478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:57.519444    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:17:57.519655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:57.519507    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:17:57.868376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:57.863780    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:17:58.520099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:58.520064    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:17:58.520657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:58.520615    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:17:59.421485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.421460    5845 scope.go:122] "RemoveContainer" containerID="49ca1d1886fd2df008b82dd9579fa8ab9c71578d8fa7d82017371521d9bf8d89"
Nov 01 22:17:59.519577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.519552    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:17:59.519675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.519599    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:17:59.888059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.888022    5845 generic.go:356] "Generic (PLEG): container finished" podID="67c66cc6-05aa-449e-aa44-618406f83882" containerID="245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a" exitCode=0
Nov 01 22:17:59.888212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.888094    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerDied","Data":"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"}
Nov 01 22:17:59.888551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.888532    5845 scope.go:122] "RemoveContainer" containerID="245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"
Nov 01 22:17:59.890368 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.890335    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:59.890781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.890713    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:17:59.890857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:17:59.890788    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:00.521123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:00.520533    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 01 22:18:00.521123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:00.520594    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 01 22:18:00.897276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:00.897192    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerStarted","Data":"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"}
Nov 01 22:18:01.395803 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:01.395767    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:18:01.396604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:01.396573    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 22:18:01.396740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:01.396630    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 22:18:01.527654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:01.527592    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:18:01.529079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:01.529055    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:18:02.395379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:02.395337    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 22:18:02.395549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:02.395409    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 22:18:02.523971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:02.523937    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:18:03.395532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:03.395499    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 22:18:03.396024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:03.395566    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 22:18:04.395817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:04.395780    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 22:18:04.396288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:04.395840    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 22:18:04.877238 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:18:04.877199    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:18:05.395978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:05.395944    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 01 22:18:05.396529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:05.396025    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 01 22:18:06.400939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:06.400899    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:18:06.402101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:06.402077    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:18:15.059243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.059155    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:18:15.060331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060280    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy"
Nov 01 22:18:15.060331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060331    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy"
Nov 01 22:18:15.060548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060372    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy"
Nov 01 22:18:15.060548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060380    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy"
Nov 01 22:18:15.060548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060414    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:15.060548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060422    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:15.060767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060619    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70e5a04-ae17-4f6b-bfa7-4908166d4e33" containerName="istio-proxy"
Nov 01 22:18:15.060767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060637    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a180a956-8b70-4fba-855b-eacd78570803" containerName="istio-proxy"
Nov 01 22:18:15.060767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060653    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:15.060767 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.060661    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:15.064282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.064261    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.087656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.087609    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142342    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142424    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142460    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zch2\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142490    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142537    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142669    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142699    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142769    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.153376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.142948    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.247446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247406    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.247651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247550    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.247651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247579    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.247651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247639    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247796    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247819    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247903    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247956    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.247987    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zch2\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.248013    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.248009    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.248221    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.248486    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.248726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.248679    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.250255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.250187    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.250685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.250662    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.260404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.260381    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.260551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.260531    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zch2\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\") pod \"e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") " pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.401095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.401018    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:15.689239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.689212    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:18:15.690521 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:18:15.690495    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3574beae_8658_4d85_9d12_255ade49f917.slice/crio-4e1e72afb5e2086be7a368f2b13c528995bc0e7a07532faf59631d7174a6fbff WatchSource:0}: Error finding container 4e1e72afb5e2086be7a368f2b13c528995bc0e7a07532faf59631d7174a6fbff: Status 404 returned error can't find the container with id 4e1e72afb5e2086be7a368f2b13c528995bc0e7a07532faf59631d7174a6fbff
Nov 01 22:18:15.694838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.694797    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:15.694941 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.694886    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:15.694941 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:15.694929    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:16.020877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:16.020845    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" event={"ID":"3574beae-8658-4d85-9d12-255ade49f917","Type":"ContainerStarted","Data":"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"}
Nov 01 22:18:16.021090 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:16.020900    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" event={"ID":"3574beae-8658-4d85-9d12-255ade49f917","Type":"ContainerStarted","Data":"4e1e72afb5e2086be7a368f2b13c528995bc0e7a07532faf59631d7174a6fbff"}
Nov 01 22:18:16.401417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:16.401333    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:16.402588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:16.402556    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:16.402717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:16.402624    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:17.402238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:17.402203    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:17.402672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:17.402261    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:18.402057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:18.402023    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:18.402290 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:18.402075    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:19.401586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:19.401551    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:19.401781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:19.401615    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:20.401398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:20.401362    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:20.402034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:20.401414    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:21.407435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:21.406445    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:21.407435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:21.406508    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:22.401626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:22.401587    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Nov 01 22:18:22.401844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:22.401726    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Nov 01 22:18:23.407724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:23.407693    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:23.408247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:23.408229    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:23.408960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:23.408939    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:18:23.434791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:23.434749    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podStartSLOduration=8.434735847 podStartE2EDuration="8.434735847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:16.047537148 +0000 UTC m=+1999.303716905" watchObservedRunningTime="2026-11-01 22:18:23.434735847 +0000 UTC m=+2006.690915606"
Nov 01 22:18:34.137508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:34.137458    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:18:34.137997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:34.137535    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:18:34.862230 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:18:34.862186    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:18:38.345153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.345116    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:18:38.346439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.346413    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:38.346537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.346440    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48f8f3a-093e-43f4-84cc-cbd03315ef3f" containerName="istio-proxy"
Nov 01 22:18:38.506325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.498941    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:18:38.506325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.499075    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.548374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.546969    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.548374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.547043    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdm6\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.548374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.547113    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.548374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.547281    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.551381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.551350    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.554174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.551616    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.554174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.551704    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.554174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.551776    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.554174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.551872    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.557199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.555082    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d2832c-openshift-default-dockercfg-vhkf4\""
Nov 01 22:18:38.654541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654445    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdm6\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654538    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654718    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654775    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654801    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654853    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.654993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.654918    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.655169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.655009    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.655169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.655082    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.655663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.655635    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.656772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.656721    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.657720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.657696    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.660361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.658864    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.660361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.659191    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.664140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.663966    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.668083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.668055    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.669441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.669421    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.683313 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.677011    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdm6\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\") pod \"e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") " pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:38.911964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:38.911884    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:39.337820 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.337784    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:18:39.338909 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:18:39.338883    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366d1dd_7a69_47c6_983f_fb72ddccce54.slice/crio-768d80c42efe7f6a3d9d84b64a865731b1c30f4e73e72f4844201c7ccaa420e4 WatchSource:0}: Error finding container 768d80c42efe7f6a3d9d84b64a865731b1c30f4e73e72f4844201c7ccaa420e4: Status 404 returned error can't find the container with id 768d80c42efe7f6a3d9d84b64a865731b1c30f4e73e72f4844201c7ccaa420e4
Nov 01 22:18:39.341877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.341848    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:39.341951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.341913    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:39.341951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.341945    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:39.614307 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.614209    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerStarted","Data":"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"}
Nov 01 22:18:39.614307 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.614279    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerStarted","Data":"768d80c42efe7f6a3d9d84b64a865731b1c30f4e73e72f4844201c7ccaa420e4"}
Nov 01 22:18:39.640380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.640320    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podStartSLOduration=1.6402888930000001 podStartE2EDuration="1.640288893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:39.636890041 +0000 UTC m=+2022.893069805" watchObservedRunningTime="2026-11-01 22:18:39.640288893 +0000 UTC m=+2022.896468657"
Nov 01 22:18:39.912394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:39.912286    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:40.919329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:40.919278    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:41.639884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:41.639855    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:41.640757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:41.640735    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:18:43.200779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.200699    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:18:43.204349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.204330    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.214427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.214338    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d2832c\""
Nov 01 22:18:43.214427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.214359    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d2832c-dockercfg-7gpvn\""
Nov 01 22:18:43.216335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.216312    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:18:43.252214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.252178    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:18:43.256009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.255974    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.270861 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.270837    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:18:43.377343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.358413    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.386404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.381936    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.393327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.386903    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sg9j\" (UniqueName: \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.393743 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.393710    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pkm\" (UniqueName: \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.503583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.503523    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.503795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.503597    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.503876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.503797    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sg9j\" (UniqueName: \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.503974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.503957    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pkm\" (UniqueName: \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.507657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.507607    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.511221 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.511185    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.523660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.521902    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pkm\" (UniqueName: \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\") pod \"payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.534493 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.534100    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sg9j\" (UniqueName: \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\") pod \"payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.575097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.575066    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:43.664447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.664374    5845 generic.go:356] "Generic (PLEG): container finished" podID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerID="037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7" exitCode=0
Nov 01 22:18:43.664601 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.664485    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerDied","Data":"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"}
Nov 01 22:18:43.665061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.664987    5845 scope.go:122] "RemoveContainer" containerID="037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"
Nov 01 22:18:43.666833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.666807    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:43.666932 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.666873    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:43.666978 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.666935    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:18:43.822589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.820507    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:43.867510 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:18:43.866733    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbe1608_dc84_42dd_83b7_54be275ae505.slice/crio-feb2d485c60d8e4d97172f15bfa4dcc12cd7bb2c2effa63357de582437834070 WatchSource:0}: Error finding container feb2d485c60d8e4d97172f15bfa4dcc12cd7bb2c2effa63357de582437834070: Status 404 returned error can't find the container with id feb2d485c60d8e4d97172f15bfa4dcc12cd7bb2c2effa63357de582437834070
Nov 01 22:18:43.867802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:43.867775    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:18:44.121349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.121320    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:18:44.123371 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:18:44.123346    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd832d3b8_c3e0_4d1e_a44c_e80aca53a364.slice/crio-ac8ec4c1600ad294ff0bc3f7b8ab156a8fdc21ca70fef2b3997fbfe2d02faea6 WatchSource:0}: Error finding container ac8ec4c1600ad294ff0bc3f7b8ab156a8fdc21ca70fef2b3997fbfe2d02faea6: Status 404 returned error can't find the container with id ac8ec4c1600ad294ff0bc3f7b8ab156a8fdc21ca70fef2b3997fbfe2d02faea6
Nov 01 22:18:44.672959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.672927    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" event={"ID":"9fbe1608-dc84-42dd-83b7-54be275ae505","Type":"ContainerStarted","Data":"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"}
Nov 01 22:18:44.673439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.672978    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" event={"ID":"9fbe1608-dc84-42dd-83b7-54be275ae505","Type":"ContainerStarted","Data":"feb2d485c60d8e4d97172f15bfa4dcc12cd7bb2c2effa63357de582437834070"}
Nov 01 22:18:44.673439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.673019    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:44.674748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.674723    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" event={"ID":"d832d3b8-c3e0-4d1e-a44c-e80aca53a364","Type":"ContainerStarted","Data":"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"}
Nov 01 22:18:44.674876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.674760    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" event={"ID":"d832d3b8-c3e0-4d1e-a44c-e80aca53a364","Type":"ContainerStarted","Data":"ac8ec4c1600ad294ff0bc3f7b8ab156a8fdc21ca70fef2b3997fbfe2d02faea6"}
Nov 01 22:18:44.674876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.674798    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:44.676776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.676750    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerStarted","Data":"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"}
Nov 01 22:18:44.693242 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.693194    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podStartSLOduration=1.693182438 podStartE2EDuration="1.693182438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:44.691928306 +0000 UTC m=+2027.948108069" watchObservedRunningTime="2026-11-01 22:18:44.693182438 +0000 UTC m=+2027.949362197"
Nov 01 22:18:44.711038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:44.710984    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podStartSLOduration=1.710968737 podStartE2EDuration="1.710968737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:18:44.709690559 +0000 UTC m=+2027.965870324" watchObservedRunningTime="2026-11-01 22:18:44.710968737 +0000 UTC m=+2027.967148500"
Nov 01 22:18:45.585139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:45.585107    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:18:45.585917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:45.585887    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:45.586067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:45.585951    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:46.585081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:46.585047    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:46.586089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:46.585108    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:47.585073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:47.585037    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:47.585277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:47.585088    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:48.096446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:48.096417    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:18:48.586321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:48.585054    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:48.586321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:48.585133    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:49.585439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:49.585402    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:49.585643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:49.585471    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:50.585028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:50.584988    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 01 22:18:50.585503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:50.585070    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 01 22:18:51.590656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:51.590622    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:18:51.592596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:51.592570    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:18:55.688602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:55.688561    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:18:55.689159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:55.688740    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:18:59.755960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.755923    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:18:59.761249 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.761224    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.770116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.770090    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d2832c-serving-cert\""
Nov 01 22:18:59.777486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.777458    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:18:59.863147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.863107    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.863388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.863165    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsxb\" (UniqueName: \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.863388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.863221    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.863388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.863366    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.972170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.972135    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.972566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.972541    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsxb\" (UniqueName: \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.972723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.972703    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.973100 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.972945    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.974697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.974672    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:18:59.985252 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:18:59.985223    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:00.001323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:00.001275    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:00.001605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:00.001350    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsxb\" (UniqueName: \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\") pod \"maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:00.100744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:00.100657    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:00.398334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:00.397130    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:19:00.825688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:00.825649    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" event={"ID":"dd3e2e23-6f3d-48cb-9e35-574acedff5e0","Type":"ContainerStarted","Data":"aa7d1e1c34df956a2089d2e339cba53bcecf31142d4fd10430da709c50d849d5"}
Nov 01 22:19:01.925319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:01.916978    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" event={"ID":"dd3e2e23-6f3d-48cb-9e35-574acedff5e0","Type":"ContainerStarted","Data":"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"}
Nov 01 22:19:01.925319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:01.918237    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:02.004318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:02.000351    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" podStartSLOduration=2.432583635 podStartE2EDuration="3.00032656s" totalImagesPullingTime="567.742925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:19:01.976225151 +0000 UTC m=+2045.232404915" watchObservedRunningTime="2026-11-01 22:19:02.00032656 +0000 UTC m=+2045.256506324"
Nov 01 22:19:04.818182 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:19:04.818129    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:19:07.400755 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:07.400720    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:19:08.938152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:08.938120    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:19:22.440884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.440827    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" podUID="3f546dd9-4d51-45f6-aea1-62060513f622" containerName="main" containerID="cri-o://1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0" gracePeriod=45
Nov 01 22:19:22.791777 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.791752    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:19:22.926785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.926753    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\" (UniqueName: \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.926968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.926888    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.927030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927014    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.927178 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927156    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.927343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927181    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:19:22.927343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927225    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.927343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927270    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\") pod \"3f546dd9-4d51-45f6-aea1-62060513f622\" (UID: \"3f546dd9-4d51-45f6-aea1-62060513f622\") "
Nov 01 22:19:22.927575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927548    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:19:22.927765 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.927746    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:19:22.928342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.928317    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:22.928342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.928340    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:22.928528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.928353    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:22.929223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.929202    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "kube-api-access-5hxfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:19:22.929581 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.929553    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:19:22.929693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:22.929615    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm" pod "3f546dd9-4d51-45f6-aea1-62060513f622" (UID: "3f546dd9-4d51-45f6-aea1-62060513f622"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:19:23.030856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.030751    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxfq\" (UniqueName: \"kubernetes.io/projected/3f546dd9-4d51-45f6-aea1-62060513f622-kube-api-access-5hxfq\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:23.030856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.030792    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f546dd9-4d51-45f6-aea1-62060513f622-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:23.030856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.030809    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f546dd9-4d51-45f6-aea1-62060513f622-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:23.166775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.166711    5845 generic.go:356] "Generic (PLEG): container finished" podID="3f546dd9-4d51-45f6-aea1-62060513f622" containerID="1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0" exitCode=0
Nov 01 22:19:23.166775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.166753    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"
Nov 01 22:19:23.166993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.166803    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" event={"ID":"3f546dd9-4d51-45f6-aea1-62060513f622","Type":"ContainerDied","Data":"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"}
Nov 01 22:19:23.166993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.166849    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc" event={"ID":"3f546dd9-4d51-45f6-aea1-62060513f622","Type":"ContainerDied","Data":"4f56c935367d12cee7f31e9ed7992a18b2fee1449d8022544b30853b3c7e2286"}
Nov 01 22:19:23.166993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.166866    5845 scope.go:122] "RemoveContainer" containerID="1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"
Nov 01 22:19:23.180363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.180345    5845 scope.go:122] "RemoveContainer" containerID="1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"
Nov 01 22:19:23.180618 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:19:23.180601    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0\": container with ID starting with 1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0 not found: ID does not exist" containerID="1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"
Nov 01 22:19:23.180688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.180626    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0"} err="failed to get container status \"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0\": rpc error: code = NotFound desc = could not find container \"1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0\": container with ID starting with 1b91ffebc32b8554a142a064200d7cbc14b220e0142dcd0ebaca9a3121953ee0 not found: ID does not exist"
Nov 01 22:19:23.192350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.192327    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:19:23.196077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:23.196057    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dfdb11/test-model-dfdb11-kserve-6978b659d7-npnmc"]
Nov 01 22:19:24.865750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:24.865715    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f546dd9-4d51-45f6-aea1-62060513f622" path="/var/lib/kubelet/pods/3f546dd9-4d51-45f6-aea1-62060513f622/volumes"
Nov 01 22:19:32.348500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:32.348469    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:19:32.367335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:32.367286    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:19:34.820098 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:19:34.820044    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:19:37.349455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:37.349415    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing" containerID="cri-o://334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663" gracePeriod=25
Nov 01 22:19:37.368887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:37.368843    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing" containerID="cri-o://052764ff07a31452fb08bf7714fe8052bf1010a8c7354fe05609003e70c7f0a9" gracePeriod=25
Nov 01 22:19:38.746483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:38.746382    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:19:38.746483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:38.746408    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:19:48.746715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:48.746681    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:19:48.746715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:48.746702    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:19:57.596120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.595949    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:19:57.607690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.607639    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:19:57.802343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.753909    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:19:57.802646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.802577    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:19:57.991465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.991368    5845 generic.go:356] "Generic (PLEG): container finished" podID="40cdf802-c25f-446d-ae2b-26a71629e212" containerID="334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663" exitCode=0
Nov 01 22:19:57.992275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:57.991971    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" event={"ID":"40cdf802-c25f-446d-ae2b-26a71629e212","Type":"ContainerDied","Data":"334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663"}
Nov 01 22:19:58.048452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:58.048407    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:19:58.063548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:58.063458    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:19:58.117177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:58.087801    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 01 22:19:58.735598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:58.716372    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:19:58.774944 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:19:58.771767    5845 logging.go:55] [core] [Channel #5547 SubChannel #5549]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.132:50052", ServerName: "10.134.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.132:50052: connect: connection refused"
Nov 01 22:19:59.006513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.006481    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:19:59.006896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.006833    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:19:59.051747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.051341    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" event={"ID":"40cdf802-c25f-446d-ae2b-26a71629e212","Type":"ContainerDied","Data":"2bdea82a14ae1bb0fb3a67bd173e16d02c6a8b51c61100f03ccad9a8594f3acf"}
Nov 01 22:19:59.051747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.051415    5845 scope.go:122] "RemoveContainer" containerID="334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663"
Nov 01 22:19:59.051747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.051565    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"
Nov 01 22:19:59.059041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.059014    5845 generic.go:356] "Generic (PLEG): container finished" podID="287faada-dc3c-4cd5-97e4-492288419e8e" containerID="052764ff07a31452fb08bf7714fe8052bf1010a8c7354fe05609003e70c7f0a9" exitCode=0
Nov 01 22:19:59.059251 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.059231    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" event={"ID":"287faada-dc3c-4cd5-97e4-492288419e8e","Type":"ContainerDied","Data":"052764ff07a31452fb08bf7714fe8052bf1010a8c7354fe05609003e70c7f0a9"}
Nov 01 22:19:59.110501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.110468    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:19:59.118819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.118122    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\" (UniqueName: \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\") pod \"40cdf802-c25f-446d-ae2b-26a71629e212\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") "
Nov 01 22:19:59.118819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.118352    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\") pod \"40cdf802-c25f-446d-ae2b-26a71629e212\" (UID: \"40cdf802-c25f-446d-ae2b-26a71629e212\") "
Nov 01 22:19:59.119781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.119569    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume" pod "40cdf802-c25f-446d-ae2b-26a71629e212" (UID: "40cdf802-c25f-446d-ae2b-26a71629e212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:19:59.122389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.122359    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj" pod "40cdf802-c25f-446d-ae2b-26a71629e212" (UID: "40cdf802-c25f-446d-ae2b-26a71629e212"). InnerVolumeSpecName "kube-api-access-5jvwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:19:59.220588 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.220551    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\") pod \"287faada-dc3c-4cd5-97e4-492288419e8e\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") "
Nov 01 22:19:59.220797 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.220682    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\" (UniqueName: \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\") pod \"287faada-dc3c-4cd5-97e4-492288419e8e\" (UID: \"287faada-dc3c-4cd5-97e4-492288419e8e\") "
Nov 01 22:19:59.220985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.220957    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume" pod "287faada-dc3c-4cd5-97e4-492288419e8e" (UID: "287faada-dc3c-4cd5-97e4-492288419e8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:19:59.221663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.221629    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40cdf802-c25f-446d-ae2b-26a71629e212-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:59.221663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.221658    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287faada-dc3c-4cd5-97e4-492288419e8e-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:59.221867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.221672    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvwj\" (UniqueName: \"kubernetes.io/projected/40cdf802-c25f-446d-ae2b-26a71629e212-kube-api-access-5jvwj\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:59.223128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.223098    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8" pod "287faada-dc3c-4cd5-97e4-492288419e8e" (UID: "287faada-dc3c-4cd5-97e4-492288419e8e"). InnerVolumeSpecName "kube-api-access-kspf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:19:59.323580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.323504    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kspf8\" (UniqueName: \"kubernetes.io/projected/287faada-dc3c-4cd5-97e4-492288419e8e-kube-api-access-kspf8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:19:59.376023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.375966    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:19:59.380623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.380597    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4"]
Nov 01 22:19:59.564684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.564651    5845 scope.go:122] "RemoveContainer" containerID="052764ff07a31452fb08bf7714fe8052bf1010a8c7354fe05609003e70c7f0a9"
Nov 01 22:19:59.763973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.763936    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-dfdb11-fd9d47844-gc6g4" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.133:50052\" within 1s: context deadline exceeded"
Nov 01 22:19:59.764437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.763936    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.132:50052\" within 1s: context deadline exceeded"
Nov 01 22:19:59.764437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:19:59.764076    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:19:59.764437 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:19:59.764012    5845 logging.go:55] [core] [Channel #5546 SubChannel #5548]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.133:50052", ServerName: "10.134.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.133:50052: operation was canceled"
Nov 01 22:20:00.086732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.086686    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"
Nov 01 22:20:00.087073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.087043    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7" event={"ID":"287faada-dc3c-4cd5-97e4-492288419e8e","Type":"ContainerDied","Data":"34919abcc9f8eba32d9e892410f07ced0cd4a935085449a611568d57580742e2"}
Nov 01 22:20:00.127973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.127943    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:20:00.131446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.131419    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dfdb11-7c78bdc4dc-fkzr7"]
Nov 01 22:20:00.863333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.863284    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" path="/var/lib/kubelet/pods/287faada-dc3c-4cd5-97e4-492288419e8e/volumes"
Nov 01 22:20:00.863754 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:00.863741    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" path="/var/lib/kubelet/pods/40cdf802-c25f-446d-ae2b-26a71629e212/volumes"
Nov 01 22:20:01.185412 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:01.185330    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:04.818815 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:04.818777    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:20:09.254378 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:09.254338    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:11.216084 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:11.216042    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:14.317845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.317810    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:20:14.318710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318688    5845 cpu_manager.go:414] "containerMap: removing container" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing"
Nov 01 22:20:14.318710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318708    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318747    5845 cpu_manager.go:414] "containerMap: removing container" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318754    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318793    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3f546dd9-4d51-45f6-aea1-62060513f622" containerName="main"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318799    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f546dd9-4d51-45f6-aea1-62060513f622" containerName="main"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318921    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="287faada-dc3c-4cd5-97e4-492288419e8e" containerName="payload-pre-processing"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318934    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f546dd9-4d51-45f6-aea1-62060513f622" containerName="main"
Nov 01 22:20:14.319416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.318943    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cdf802-c25f-446d-ae2b-26a71629e212" containerName="payload-processing"
Nov 01 22:20:14.325565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.325537    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.336896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.336873    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-f758d9\"/\"kube-root-ca.crt\""
Nov 01 22:20:14.337038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.336903    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-f758d9\"/\"default-dockercfg-hgdf2\""
Nov 01 22:20:14.337147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.336963    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-f758d9\"/\"openshift-service-ca.crt\""
Nov 01 22:20:14.337147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.337056    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-f758d9\"/\"rate-test-model-f758d9-kserve-self-signed-certs\""
Nov 01 22:20:14.338217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.338169    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:20:14.416564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.416537    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.416710 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.416693    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdwr\" (UniqueName: \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.416757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.416739    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.416895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.416880    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.416998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.416981    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.417071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.417058    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523436    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523572    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523676    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523708    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523843    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdwr\" (UniqueName: \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.523894    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.524476    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.524792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.524734    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.528356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.525391    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.548899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.548869    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.550025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.549999    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.564895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.564843    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdwr\" (UniqueName: \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\") pod \"rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") " pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.649052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.648962    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:14.937727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:14.937700    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:20:14.938663 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:20:14.938638    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a0896c_8c14_47ac_96fa_4b0fe5f8d7d8.slice/crio-f748c3adb2337bf96d6bc24d3d2789148f23ccab37e70f9cd4439ec491bebd4c WatchSource:0}: Error finding container f748c3adb2337bf96d6bc24d3d2789148f23ccab37e70f9cd4439ec491bebd4c: Status 404 returned error can't find the container with id f748c3adb2337bf96d6bc24d3d2789148f23ccab37e70f9cd4439ec491bebd4c
Nov 01 22:20:15.283214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:15.282563    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" event={"ID":"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8","Type":"ContainerStarted","Data":"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"}
Nov 01 22:20:15.283214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:15.282639    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" event={"ID":"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8","Type":"ContainerStarted","Data":"f748c3adb2337bf96d6bc24d3d2789148f23ccab37e70f9cd4439ec491bebd4c"}
Nov 01 22:20:15.307361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:15.307258    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podStartSLOduration=1.30723477 podStartE2EDuration="1.30723477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:20:15.30550928 +0000 UTC m=+2118.561689045" watchObservedRunningTime="2026-11-01 22:20:15.30723477 +0000 UTC m=+2118.563414535"
Nov 01 22:20:15.649679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:15.649599    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:18.081133 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:18.081097    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:18.081514 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:18.081151    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:21.232586 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:21.232546    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:24.159197 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:24.159149    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:24.652321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:24.650593    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:24.673045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:24.673012    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:25.383369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:25.383340    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:20:31.332899 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:31.332850    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:34.832639 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:34.832579    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:20:39.231314 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:39.231259    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:41.352533 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:41.352494    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:42.681450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:42.681413    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:20:42.681939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:42.681481    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:20:42.831435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:42.831402    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:20:42.831631 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:42.831481    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:20:51.461148 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:51.461104    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:54.158372 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:20:54.158332    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdf802_c25f_446d_ae2b_26a71629e212.slice/crio-conmon-334979659e2cd3569d8a829f7f93994bdd1c26035956ed3093f8bd9236d98663.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:20:57.681454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:57.681412    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:20:57.682034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:57.681479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:20:57.831423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:57.831387    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:20:57.835867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:20:57.831455    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:21:04.043550 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.043514    5845 generic.go:356] "Generic (PLEG): container finished" podID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerID="f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826" exitCode=0
Nov 01 22:21:04.044195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.043593    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerDied","Data":"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"}
Nov 01 22:21:04.064382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.064358    5845 scope.go:122] "RemoveContainer" containerID="f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"
Nov 01 22:21:04.260111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.197504    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:04.260111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.197591    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:04.260111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:04.197627    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:05.053930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:05.053896    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerStarted","Data":"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"}
Nov 01 22:21:05.176843 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:05.176798    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:21:05.912414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:05.912375    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:21:05.912722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:05.912691    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:05.966470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:05.912763    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:06.913099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:06.913059    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:06.913614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:06.913131    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:07.088020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.087990    5845 generic.go:356] "Generic (PLEG): container finished" podID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerID="6efd6b77be58641f83b2c999b9d033877be00225997d0bedafb510ce54e4f838" exitCode=0
Nov 01 22:21:07.088196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.088067    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerDied","Data":"6efd6b77be58641f83b2c999b9d033877be00225997d0bedafb510ce54e4f838"}
Nov 01 22:21:07.088855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.088574    5845 scope.go:122] "RemoveContainer" containerID="6efd6b77be58641f83b2c999b9d033877be00225997d0bedafb510ce54e4f838"
Nov 01 22:21:07.090617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.090593    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:07.090700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.090669    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:07.090738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.090703    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:07.914178 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.913541    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:07.914178 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:07.913604    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:08.097633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.097599    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerStarted","Data":"3846430b5c367ad4fae42684a8cc92b6e7441b3949aacd42e753d4b4d6095097"}
Nov 01 22:21:08.796331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.794387    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:21:08.796331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.795013    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:08.796331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.795064    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:08.917321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.913702    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:08.917321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:08.913775    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:09.798322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:09.795260    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:09.798322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:09.795345    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:09.912497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:09.912461    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:09.912683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:09.912513    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:10.795686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:10.795628    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:10.796357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:10.796272    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:10.912258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:10.912220    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:21:10.912471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:10.912270    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 01 22:21:11.640190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:11.640151    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:21:11.797010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:11.789717    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:21:11.797010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:11.792084    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:11.797010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:11.792149    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:11.920643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:11.920562    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:21:12.128350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:12.128288    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:21:12.792025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:12.791987    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:12.792234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:12.792046    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:13.792813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:13.792778    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 01 22:21:13.793215 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:13.792842    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 01 22:21:14.797997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:14.797964    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:21:14.799427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:14.799402    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:21:20.357689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:20.357651    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 22:21:20.358089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:20.357902    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" containerID="cri-o://74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c" gracePeriod=30
Nov 01 22:21:24.166746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:24.166707    5845 patch_prober.go:29] interesting pod/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:21:24.167134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:24.166779    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:21:25.727555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.727531    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 22:21:25.817022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.816992    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.817218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817032    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.817218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817088    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.817218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817181    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817261    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817350    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817422    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817482    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817530    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\") pod \"48dba715-6fc8-428c-9f1a-1febdabc712e\" (UID: \"48dba715-6fc8-428c-9f1a-1febdabc712e\") "
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817833    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.817974    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818053    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818088    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818509    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818526    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dba715-6fc8-428c-9f1a-1febdabc712e-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818535    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.818706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.818544    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.822619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.819008    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:25.827944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.827918    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:25.828047 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.827941    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:21:25.828047 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.828005    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:21:25.828141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.828046    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46" pod "48dba715-6fc8-428c-9f1a-1febdabc712e" (UID: "48dba715-6fc8-428c-9f1a-1febdabc712e"). InnerVolumeSpecName "kube-api-access-qzd46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:21:25.920836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.920803    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.920836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.920838    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.921402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.920855    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.921402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.920873    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzd46\" (UniqueName: \"kubernetes.io/projected/48dba715-6fc8-428c-9f1a-1febdabc712e-kube-api-access-qzd46\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:25.921402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:25.920891    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dba715-6fc8-428c-9f1a-1febdabc712e-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:26.246576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.246545    5845 generic.go:356] "Generic (PLEG): container finished" podID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerID="74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c" exitCode=0
Nov 01 22:21:26.246788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.246597    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"
Nov 01 22:21:26.246788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.246625    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerDied","Data":"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"}
Nov 01 22:21:26.247370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.247280    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p" event={"ID":"48dba715-6fc8-428c-9f1a-1febdabc712e","Type":"ContainerDied","Data":"6c075b8db23893027867546e588ebcdcf8400f436b147bd6831e3715e2fea83a"}
Nov 01 22:21:26.247561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.247411    5845 scope.go:122] "RemoveContainer" containerID="74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"
Nov 01 22:21:26.265878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.265860    5845 scope.go:122] "RemoveContainer" containerID="761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"
Nov 01 22:21:26.279882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.279857    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 22:21:26.280129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.280100    5845 scope.go:122] "RemoveContainer" containerID="74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"
Nov 01 22:21:26.280434 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:26.280412    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c\": container with ID starting with 74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c not found: ID does not exist" containerID="74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"
Nov 01 22:21:26.280538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.280441    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c"} err="failed to get container status \"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c\": rpc error: code = NotFound desc = could not find container \"74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c\": container with ID starting with 74680cef60af150f460feef9fa2e1a4584d54c3807918e9c028f07479d64187c not found: ID does not exist"
Nov 01 22:21:26.280538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.280457    5845 scope.go:122] "RemoveContainer" containerID="761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"
Nov 01 22:21:26.280742 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:26.280723    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317\": container with ID starting with 761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317 not found: ID does not exist" containerID="761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"
Nov 01 22:21:26.281723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.280748    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317"} err="failed to get container status \"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317\": rpc error: code = NotFound desc = could not find container \"761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317\": container with ID starting with 761c4f57de489954b8948d9045e43a2807b13241e99d559a582095f01b4d6317 not found: ID does not exist"
Nov 01 22:21:26.283939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.283903    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b0b7a3-openshift-default-555bd95f85-zzt5p"]
Nov 01 22:21:26.867091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:26.867057    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" path="/var/lib/kubelet/pods/48dba715-6fc8-428c-9f1a-1febdabc712e/volumes"
Nov 01 22:21:34.820725 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:34.820686    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:21:49.348579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.348545    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:21:49.349377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349358    5845 cpu_manager.go:414] "containerMap: removing container" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.349377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349376    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.349531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349391    5845 cpu_manager.go:414] "containerMap: removing container" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.349531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349396    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.349627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349614    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.349682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.349633    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dba715-6fc8-428c-9f1a-1febdabc712e" containerName="istio-proxy"
Nov 01 22:21:49.354351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.354331    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.370266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.370238    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:21:49.429125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429063    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429306    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429413    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429464    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429548    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x88x\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429626    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429702    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429740    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.429836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.429789    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531313    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531380    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531405    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531436    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531483    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531611    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531721    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531742    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531767    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531850    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.531919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531919    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.535363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.531853    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x88x\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.535363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.532052    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.535363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.532436    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.535363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.535153    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.535591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.535377    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.543484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.543459    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.544144 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.543731    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x88x\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:49.671784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:49.671696    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:50.012421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.012380    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:21:50.023250 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:21:50.023216    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf934630d_e314_4d71_90f1_539f20f97241.slice/crio-54b66b2364098a16c3a68fd326cd65577f19b62e71c9f721f0d76759b032f0fd WatchSource:0}: Error finding container 54b66b2364098a16c3a68fd326cd65577f19b62e71c9f721f0d76759b032f0fd: Status 404 returned error can't find the container with id 54b66b2364098a16c3a68fd326cd65577f19b62e71c9f721f0d76759b032f0fd
Nov 01 22:21:50.026088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.026052    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:50.026184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.026125    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:50.026184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.026166    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:21:50.515655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.515595    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" event={"ID":"f934630d-e314-4d71-90f1-539f20f97241","Type":"ContainerStarted","Data":"d3a51cd59a2bc2c8829a376eb955dc8d1cecc88689840331cdaf0858bd311700"}
Nov 01 22:21:50.516165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.515679    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" event={"ID":"f934630d-e314-4d71-90f1-539f20f97241","Type":"ContainerStarted","Data":"54b66b2364098a16c3a68fd326cd65577f19b62e71c9f721f0d76759b032f0fd"}
Nov 01 22:21:50.539226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.539167    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podStartSLOduration=1.539148314 podStartE2EDuration="1.539148314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:21:50.538925432 +0000 UTC m=+2213.795105196" watchObservedRunningTime="2026-11-01 22:21:50.539148314 +0000 UTC m=+2213.795328078"
Nov 01 22:21:50.645996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.645965    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 22:21:50.646171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.646151    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" containerID="cri-o://c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c" gracePeriod=30
Nov 01 22:21:50.672462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:50.672435    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:51.673884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:51.673804    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:51.673884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:51.673863    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:52.681371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:52.679714    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:52.681371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:52.679813    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:53.674623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:53.672132    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:53.674623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:53.672196    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:54.228199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:54.228161    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:21:54.228792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:54.228760    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:21:54.672714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:54.672609    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:54.672714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:54.672688    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:55.678147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:55.674942    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:55.678147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:55.674997    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:56.071526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.071146    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099424    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099602    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099697    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099746    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099775    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099873    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099933    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.099988    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.101032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.100060    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\") pod \"b8681913-b53a-4ac8-8dd0-d5a239356972\" (UID: \"b8681913-b53a-4ac8-8dd0-d5a239356972\") "
Nov 01 22:21:56.115050 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.101836    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:21:56.115050 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.114626    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:56.115794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.115402    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:56.115939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.115861    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:56.147689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.138398    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:56.169680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.166520    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:21:56.177894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.177861    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:21:56.182198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.182142    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "kube-api-access-2v2qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:21:56.183453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.182452    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy" pod "b8681913-b53a-4ac8-8dd0-d5a239356972" (UID: "b8681913-b53a-4ac8-8dd0-d5a239356972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209684    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209721    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209735    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209747    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209759    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209770    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209783    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2qm\" (UniqueName: \"kubernetes.io/projected/b8681913-b53a-4ac8-8dd0-d5a239356972-kube-api-access-2v2qm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209795    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8681913-b53a-4ac8-8dd0-d5a239356972-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.211079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.209807    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8681913-b53a-4ac8-8dd0-d5a239356972-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:21:56.683355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.680698    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 01 22:21:56.683355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.680760    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 01 22:21:56.690849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.684710    5845 generic.go:356] "Generic (PLEG): container finished" podID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerID="c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c" exitCode=0
Nov 01 22:21:56.690849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.684786    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerDied","Data":"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"}
Nov 01 22:21:56.690849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.684858    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52" event={"ID":"b8681913-b53a-4ac8-8dd0-d5a239356972","Type":"ContainerDied","Data":"b31a8ee53ebc9f4e80fe1a2059172de411b8b831427cb27be82ed29df8bc2a4c"}
Nov 01 22:21:56.690849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.684896    5845 scope.go:122] "RemoveContainer" containerID="c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"
Nov 01 22:21:56.690849 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.685128    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"
Nov 01 22:21:56.719230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.719203    5845 scope.go:122] "RemoveContainer" containerID="e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"
Nov 01 22:21:56.740105 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.740064    5845 scope.go:122] "RemoveContainer" containerID="c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"
Nov 01 22:21:56.740779 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:56.740751    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c\": container with ID starting with c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c not found: ID does not exist" containerID="c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"
Nov 01 22:21:56.740865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.740796    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c"} err="failed to get container status \"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c\": rpc error: code = NotFound desc = could not find container \"c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c\": container with ID starting with c7ac15d297e546a013273c6aff495fc641131e79c28af2b7838d21339251901c not found: ID does not exist"
Nov 01 22:21:56.740865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.740841    5845 scope.go:122] "RemoveContainer" containerID="e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"
Nov 01 22:21:56.741817 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:21:56.741358    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d\": container with ID starting with e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d not found: ID does not exist" containerID="e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"
Nov 01 22:21:56.741817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.741406    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d"} err="failed to get container status \"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d\": rpc error: code = NotFound desc = could not find container \"e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d\": container with ID starting with e19a186c9f7cb2a46fb583443b10b71e94d554a5857b5d565bd4f3ae7399270d not found: ID does not exist"
Nov 01 22:21:56.752319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.747943    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 22:21:56.752319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.751954    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-4ff52"]
Nov 01 22:21:56.869344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:56.869310    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" path="/var/lib/kubelet/pods/b8681913-b53a-4ac8-8dd0-d5a239356972/volumes"
Nov 01 22:21:57.678461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:57.678417    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:57.678786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:57.678759    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:21:57.680124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:21:57.680103    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:22:04.818583 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:22:04.818544    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:22:05.404316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:05.404276    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 22:22:05.404548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:05.404514    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy" containerID="cri-o://f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba" gracePeriod=30
Nov 01 22:22:05.453150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:05.453113    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 22:22:05.453462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:05.453402    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy" containerID="cri-o://eab42bcbabf7447de6f73c88712667268eb4ace915f644aedc2c911f5bfec3bc" gracePeriod=30
Nov 01 22:22:10.811988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.811955    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 22:22:10.814753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.814722    5845 generic.go:356] "Generic (PLEG): container finished" podID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerID="f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba" exitCode=0
Nov 01 22:22:10.814902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.814754    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" event={"ID":"9d537ecc-748c-402e-9f5d-dd6855d93a88","Type":"ContainerDied","Data":"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"}
Nov 01 22:22:10.814902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.814806    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr" event={"ID":"9d537ecc-748c-402e-9f5d-dd6855d93a88","Type":"ContainerDied","Data":"d758157ffb047469e2ba19843744a2910802454ee3706321ea89ee0b1b864c21"}
Nov 01 22:22:10.814902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.814838    5845 scope.go:122] "RemoveContainer" containerID="f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"
Nov 01 22:22:10.817584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.817557    5845 generic.go:356] "Generic (PLEG): container finished" podID="42001214-96ab-4dc5-a887-cc32182431b2" containerID="eab42bcbabf7447de6f73c88712667268eb4ace915f644aedc2c911f5bfec3bc" exitCode=0
Nov 01 22:22:10.817862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.817739    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerDied","Data":"eab42bcbabf7447de6f73c88712667268eb4ace915f644aedc2c911f5bfec3bc"}
Nov 01 22:22:10.817862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.817779    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 22:22:10.817862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.817804    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm" event={"ID":"42001214-96ab-4dc5-a887-cc32182431b2","Type":"ContainerDied","Data":"298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1"}
Nov 01 22:22:10.817862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.817830    5845 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298a78bed2362def4f2e1f2c2cebb24d94d5bba274332dfa371a1a83b4f2c5c1"
Nov 01 22:22:10.830437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.830418    5845 scope.go:122] "RemoveContainer" containerID="f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"
Nov 01 22:22:10.830744 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:22:10.830715    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba\": container with ID starting with f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba not found: ID does not exist" containerID="f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"
Nov 01 22:22:10.830832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.830754    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba"} err="failed to get container status \"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba\": rpc error: code = NotFound desc = could not find container \"f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba\": container with ID starting with f7b5cf23c4a8415dd64e26c088ffeb642bb2e028b51380323cfe8e751381bbba not found: ID does not exist"
Nov 01 22:22:10.830832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.830776    5845 scope.go:122] "RemoveContainer" containerID="426b02b312f91a5b1c5c2644c712e28130f14722a9141aad0cd9f2150dccd1c2"
Nov 01 22:22:10.977084 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977057    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977129    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977223    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977257    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977386    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977437    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977493    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977562    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977623    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977679    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977684    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977699    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977809    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977837    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977849    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977937    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.977986    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978015    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978027    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978062    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978080    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978133    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\") pod \"9d537ecc-748c-402e-9f5d-dd6855d93a88\" (UID: \"9d537ecc-748c-402e-9f5d-dd6855d93a88\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978178    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\") pod \"42001214-96ab-4dc5-a887-cc32182431b2\" (UID: \"42001214-96ab-4dc5-a887-cc32182431b2\") "
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978257    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.978462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.978357    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979238    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d537ecc-748c-402e-9f5d-dd6855d93a88-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979261    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979277    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979290    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42001214-96ab-4dc5-a887-cc32182431b2-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979323    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979339    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979354    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979610    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.979869    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "kube-api-access-j6g4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980084    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980753    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980762    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980773    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980807    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.980956    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.981234    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.981818    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token" pod "42001214-96ab-4dc5-a887-cc32182431b2" (UID: "42001214-96ab-4dc5-a887-cc32182431b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:22:10.982157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:10.981842    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq" pod "9d537ecc-748c-402e-9f5d-dd6855d93a88" (UID: "9d537ecc-748c-402e-9f5d-dd6855d93a88"). InnerVolumeSpecName "kube-api-access-vq7pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:22:11.080702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080667    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080699    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080712    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080721    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080729    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080742    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6g4h\" (UniqueName: \"kubernetes.io/projected/42001214-96ab-4dc5-a887-cc32182431b2-kube-api-access-j6g4h\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080751    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080760    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42001214-96ab-4dc5-a887-cc32182431b2-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080768    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42001214-96ab-4dc5-a887-cc32182431b2-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080780    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7pq\" (UniqueName: \"kubernetes.io/projected/9d537ecc-748c-402e-9f5d-dd6855d93a88-kube-api-access-vq7pq\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.080896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.080793    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d537ecc-748c-402e-9f5d-dd6855d93a88-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:22:11.827535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.827507    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"
Nov 01 22:22:11.829500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.829475    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"
Nov 01 22:22:11.875703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.875673    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 22:22:11.879225 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.879201    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f3a6b-openshift-default-db74bcc99-s2xvm"]
Nov 01 22:22:11.887792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.887768    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 22:22:11.891064 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:11.891043    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0def36-openshift-default-7d684697cc-fdxrr"]
Nov 01 22:22:12.863405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:12.863365    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42001214-96ab-4dc5-a887-cc32182431b2" path="/var/lib/kubelet/pods/42001214-96ab-4dc5-a887-cc32182431b2/volumes"
Nov 01 22:22:12.864107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:12.864084    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" path="/var/lib/kubelet/pods/9d537ecc-748c-402e-9f5d-dd6855d93a88/volumes"
Nov 01 22:22:21.183254 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.183220    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:22:21.184023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184005    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:21.184072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184025    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:21.184072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184043    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:21.184072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184048    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:21.184072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184058    5845 cpu_manager.go:414] "containerMap: removing container" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:21.184072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184064    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184081    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184087    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184111    5845 cpu_manager.go:414] "containerMap: removing container" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184116    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184219    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184228    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:21.184272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.184236    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d537ecc-748c-402e-9f5d-dd6855d93a88" containerName="istio-proxy"
Nov 01 22:22:21.187999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.187973    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.195961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.195285    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-da34ca-openshift-default-dockercfg-2gmfx\""
Nov 01 22:22:21.200716 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.200687    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:22:21.360067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.360021    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswc7\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.360549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.360499    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.360931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.360893    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.361120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361101    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.363320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361275    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.363320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361336    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.363320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361553    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.363320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361591    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.363320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.361634    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474367    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474428    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474575    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474610    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474638    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474714    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswc7\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474805    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.474959    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.475003    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.476456    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.476755    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.477502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.476990    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.481264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.477917    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.481264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.479883    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.481264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.480592    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.486895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.486821    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.489886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.489862    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.492293 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.492271    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswc7\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\") pod \"e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") " pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.510929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.510905    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:21.812931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.812900    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:22:21.814574 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:22:21.814544    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce9762c_ab3f_434e_ad01_db18977833d3.slice/crio-921a0a5af42c0ec40a98332d593ac7a03ddd8a33ea89c15783ebf952c02a4e83 WatchSource:0}: Error finding container 921a0a5af42c0ec40a98332d593ac7a03ddd8a33ea89c15783ebf952c02a4e83: Status 404 returned error can't find the container with id 921a0a5af42c0ec40a98332d593ac7a03ddd8a33ea89c15783ebf952c02a4e83
Nov 01 22:22:21.817419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.817386    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:21.817514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.817467    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:21.817576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.817515    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:21.906079 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.906048    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" event={"ID":"fce9762c-ab3f-434e-ad01-db18977833d3","Type":"ContainerStarted","Data":"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"}
Nov 01 22:22:21.906204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:21.906104    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" event={"ID":"fce9762c-ab3f-434e-ad01-db18977833d3","Type":"ContainerStarted","Data":"921a0a5af42c0ec40a98332d593ac7a03ddd8a33ea89c15783ebf952c02a4e83"}
Nov 01 22:22:22.512130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:22.512097    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:22.513790 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:22.513763    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:22.513912 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:22.513811    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:23.511372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:23.511336    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:23.511584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:23.511396    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:24.511744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:24.511709    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:24.512143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:24.511761    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:25.515322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:25.514438    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:25.515322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:25.514505    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:26.518695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:26.518656    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:26.522092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:26.522054    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:27.511540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:27.511498    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:27.511741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:27.511570    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:28.511362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:28.511327    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Nov 01 22:22:28.511850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:28.511387    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Nov 01 22:22:29.517025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:29.516993    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:29.517530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:29.517467    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:29.518114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:29.518093    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:22:29.542113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:29.542070    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podStartSLOduration=8.542052592 podStartE2EDuration="8.542052592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:21.932508404 +0000 UTC m=+2245.188688168" watchObservedRunningTime="2026-11-01 22:22:29.542052592 +0000 UTC m=+2252.798232356"
Nov 01 22:22:34.853102 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:22:34.853065    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:22:50.504876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.504841    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:22:50.506265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.506234    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="42001214-96ab-4dc5-a887-cc32182431b2" containerName="istio-proxy"
Nov 01 22:22:50.506475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.506286    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8681913-b53a-4ac8-8dd0-d5a239356972" containerName="istio-proxy"
Nov 01 22:22:50.513612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.513593    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.520689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.520665    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:22:50.521802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.521775    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e25317-openshift-default-dockercfg-v8cbj\""
Nov 01 22:22:50.584118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584080    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.584337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584222    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.584337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584259    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnt2\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.584542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584430    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.584542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584466    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.584542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584503    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.585715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584569    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.585715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584658    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.585715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.584676    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686557    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686611    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686680    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686806    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686856    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnt2\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.686960    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.687008    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.687065    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.687099    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.687639    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.689323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.688270    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.715319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.708227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.715319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.708538    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.719322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.715955    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.745890 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.745851    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.760414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.746959    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnt2\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.767737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.767701    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.768063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.768045    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:50.831790 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:50.831755    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:51.148604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.148574    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:22:51.149535 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:22:51.149514    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e13660_8961_418b_afd7_1bdf2ee53d34.slice/crio-985f7ddbb21abc0914082c03e51cd5d68cd345ea5947d4f82f84fde096e66f50 WatchSource:0}: Error finding container 985f7ddbb21abc0914082c03e51cd5d68cd345ea5947d4f82f84fde096e66f50: Status 404 returned error can't find the container with id 985f7ddbb21abc0914082c03e51cd5d68cd345ea5947d4f82f84fde096e66f50
Nov 01 22:22:51.152266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.152231    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:51.152378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.152337    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:51.152378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.152372    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:22:51.250576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.250546    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerStarted","Data":"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"}
Nov 01 22:22:51.250719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.250598    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerStarted","Data":"985f7ddbb21abc0914082c03e51cd5d68cd345ea5947d4f82f84fde096e66f50"}
Nov 01 22:22:51.832746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:51.832716    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:52.839748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:52.839711    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:52.866946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:52.866897    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podStartSLOduration=2.866883187 podStartE2EDuration="2.866883187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:51.278557933 +0000 UTC m=+2274.534737698" watchObservedRunningTime="2026-11-01 22:22:52.866883187 +0000 UTC m=+2276.123062949"
Nov 01 22:22:53.308236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:53.308210    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:53.309118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:53.309097    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:22:54.866481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.866450    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:22:54.871142 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.871121    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:54.877462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.877381    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:22:54.877622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.877603    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e25317\""
Nov 01 22:22:54.887405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.887382    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e25317-dockercfg-gptxp\""
Nov 01 22:22:54.897359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.897329    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:22:54.910582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.910559    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:54.910971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.910941    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:22:54.956897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.956867    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 22:22:54.957353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.957318    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy" containerID="cri-o://37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4" gracePeriod=30
Nov 01 22:22:54.966603 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.966549    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls8c\" (UniqueName: \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:54.966794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.966694    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:54.966794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.966741    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8qp\" (UniqueName: \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:54.967753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:54.966888    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:55.068943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.068906    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls8c\" (UniqueName: \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:55.069113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.069001    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:55.069113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.069028    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8qp\" (UniqueName: \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:55.069113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.069111    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:55.070475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.070448    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:55.071569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.070550    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:55.084643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.084617    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8qp\" (UniqueName: \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\") pod \"payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:55.084953 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.084931    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls8c\" (UniqueName: \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\") pod \"payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:55.202422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.199200    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:55.237319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:55.236611    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:56.541320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:56.536343    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:22:56.543411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:56.543380    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:22:56.578315 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:56.578202    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" event={"ID":"7daef227-30bb-4da9-b33a-6c0ae9606e0c","Type":"ContainerStarted","Data":"a222e1a455799fcaaaa9585f0f7d59cecfa3f6b678c8663711d525c77f08a7f1"}
Nov 01 22:22:56.580394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:56.580312    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" event={"ID":"ae3ff83b-25ff-4150-9cf2-1329d4b664e1","Type":"ContainerStarted","Data":"e719bdf10ac54c0c263f3f4128de111c9133e40002c5cdcd4b0ea18b5ff2adb6"}
Nov 01 22:22:57.592399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.592278    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" event={"ID":"7daef227-30bb-4da9-b33a-6c0ae9606e0c","Type":"ContainerStarted","Data":"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"}
Nov 01 22:22:57.592399 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.592365    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:22:57.594996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.594967    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" event={"ID":"ae3ff83b-25ff-4150-9cf2-1329d4b664e1","Type":"ContainerStarted","Data":"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"}
Nov 01 22:22:57.595547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.595458    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:22:57.614508 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.614455    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" podStartSLOduration=3.614437412 podStartE2EDuration="3.614437412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:57.611620649 +0000 UTC m=+2280.867800413" watchObservedRunningTime="2026-11-01 22:22:57.614437412 +0000 UTC m=+2280.870617175"
Nov 01 22:22:57.632155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:57.632106    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" podStartSLOduration=3.632094548 podStartE2EDuration="3.632094548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:57.630410293 +0000 UTC m=+2280.886590059" watchObservedRunningTime="2026-11-01 22:22:57.632094548 +0000 UTC m=+2280.888274310"
Nov 01 22:22:59.695908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:22:59.695875    5845 scope.go:122] "RemoveContainer" containerID="eab42bcbabf7447de6f73c88712667268eb4ace915f644aedc2c911f5bfec3bc"
Nov 01 22:23:00.271831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.271803    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 22:23:00.380582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380548    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.380742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380643    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.380742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380702    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.380742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380727    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.380889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380765    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.380889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380846    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.381002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380887    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:00.381002 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.380970    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.381122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381000    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.381238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381220    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\") pod \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\" (UID: \"59b5e520-f8b0-41b6-91a9-7b030654b9e4\") "
Nov 01 22:23:00.381336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381267    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:23:00.381593 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381567    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:00.381792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381702    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:00.381792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.381779    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:00.382184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.382155    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.382276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.382185    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.382276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.382200    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.382276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.382214    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.382276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.382228    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.383388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.383363    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "kube-api-access-ldsp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:00.383495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.383457    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:23:00.384650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.383583    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:00.384650 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.383652    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token" pod "59b5e520-f8b0-41b6-91a9-7b030654b9e4" (UID: "59b5e520-f8b0-41b6-91a9-7b030654b9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:00.483956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.483924    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.483956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.483954    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldsp9\" (UniqueName: \"kubernetes.io/projected/59b5e520-f8b0-41b6-91a9-7b030654b9e4-kube-api-access-ldsp9\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.484177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.483966    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.484177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.483976    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b5e520-f8b0-41b6-91a9-7b030654b9e4-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:00.620925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.620839    5845 generic.go:356] "Generic (PLEG): container finished" podID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerID="37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4" exitCode=0
Nov 01 22:23:00.620925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.620887    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"
Nov 01 22:23:00.621112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.620934    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" event={"ID":"59b5e520-f8b0-41b6-91a9-7b030654b9e4","Type":"ContainerDied","Data":"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"}
Nov 01 22:23:00.621112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.621018    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh" event={"ID":"59b5e520-f8b0-41b6-91a9-7b030654b9e4","Type":"ContainerDied","Data":"82a4a37194d45dfae56863b87012ce8c4dae39dd0438f6b795893a36733b32e1"}
Nov 01 22:23:00.621112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.621050    5845 scope.go:122] "RemoveContainer" containerID="37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"
Nov 01 22:23:00.633806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.633788    5845 scope.go:122] "RemoveContainer" containerID="37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"
Nov 01 22:23:00.634074 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:00.634054    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4\": container with ID starting with 37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4 not found: ID does not exist" containerID="37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"
Nov 01 22:23:00.634140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.634087    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4"} err="failed to get container status \"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4\": rpc error: code = NotFound desc = could not find container \"37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4\": container with ID starting with 37b4f34b006c07cfd8a711a32da20a0ee148147ee14724dbd2f487f982459de4 not found: ID does not exist"
Nov 01 22:23:00.650674 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.650647    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 22:23:00.654408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.654387    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-5wxxh"]
Nov 01 22:23:00.863585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:00.863553    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" path="/var/lib/kubelet/pods/59b5e520-f8b0-41b6-91a9-7b030654b9e4/volumes"
Nov 01 22:23:04.818819 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:04.818773    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:23:08.606967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:08.606908    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:23:08.607375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:08.607016    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:23:34.061322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:34.056906    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:23:34.061322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:34.057172    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy" containerID="cri-o://57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf" gracePeriod=30
Nov 01 22:23:34.818809 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:34.818774    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:23:39.123018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.122987    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:23:39.140135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.140095    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:23:39.666351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.656761    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:23:39.797372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797338    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.797546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797391    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.797546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797479    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797547    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797610    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797642    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797661    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797661    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797695    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797747    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797883    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.797974    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\") pod \"3f10c1d2-c233-4174-939b-676ec58c594d\" (UID: \"3f10c1d2-c233-4174-939b-676ec58c594d\") "
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798115    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:23:39.798522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798292    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:39.799175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798691    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.799175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798715    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.799175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798729    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f10c1d2-c233-4174-939b-676ec58c594d-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.799175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798745    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.799175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.798787    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.800794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.800766    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:39.801618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.801536    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:23:39.802099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.802046    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "kube-api-access-wqdmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:39.802314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.802274    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo" pod "3f10c1d2-c233-4174-939b-676ec58c594d" (UID: "3f10c1d2-c233-4174-939b-676ec58c594d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:23:39.900847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.900763    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f10c1d2-c233-4174-939b-676ec58c594d-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.900847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.900790    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.900847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.900800    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f10c1d2-c233-4174-939b-676ec58c594d-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:39.900847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:39.900811    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqdmb\" (UniqueName: \"kubernetes.io/projected/3f10c1d2-c233-4174-939b-676ec58c594d-kube-api-access-wqdmb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:40.023147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.023111    5845 generic.go:356] "Generic (PLEG): container finished" podID="3f10c1d2-c233-4174-939b-676ec58c594d" containerID="57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf" exitCode=0
Nov 01 22:23:40.023359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.023159    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"
Nov 01 22:23:40.023359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.023187    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerDied","Data":"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"}
Nov 01 22:23:40.023359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.023246    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z" event={"ID":"3f10c1d2-c233-4174-939b-676ec58c594d","Type":"ContainerDied","Data":"cc331a195834ae8b4fe255ccf24cc38bce285333dfd4d821b609ba272c4c29f7"}
Nov 01 22:23:40.023359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.023272    5845 scope.go:122] "RemoveContainer" containerID="57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"
Nov 01 22:23:40.042698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.038204    5845 scope.go:122] "RemoveContainer" containerID="4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"
Nov 01 22:23:40.058347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.053233    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:23:40.058347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.057552    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7e2df-openshift-default-584f54d89c-pwp7z"]
Nov 01 22:23:40.058641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.058361    5845 scope.go:122] "RemoveContainer" containerID="57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"
Nov 01 22:23:40.059975 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:40.059571    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf\": container with ID starting with 57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf not found: ID does not exist" containerID="57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"
Nov 01 22:23:40.059975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.059603    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf"} err="failed to get container status \"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf\": rpc error: code = NotFound desc = could not find container \"57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf\": container with ID starting with 57fa7e01b14f1765626b3c1bac2bc8c05612b5fff00ae83e4c44774a9ce163bf not found: ID does not exist"
Nov 01 22:23:40.059975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.059621    5845 scope.go:122] "RemoveContainer" containerID="4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"
Nov 01 22:23:40.060218 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:40.060112    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701\": container with ID starting with 4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701 not found: ID does not exist" containerID="4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"
Nov 01 22:23:40.060218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.060143    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701"} err="failed to get container status \"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701\": rpc error: code = NotFound desc = could not find container \"4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701\": container with ID starting with 4d849297875d75561f68f176915bcc6a1f372d883a9e61c0d43df7c2dab27701 not found: ID does not exist"
Nov 01 22:23:40.863589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:40.863558    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" path="/var/lib/kubelet/pods/3f10c1d2-c233-4174-939b-676ec58c594d/volumes"
Nov 01 22:23:44.124086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.124047    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" podUID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" containerName="payload-processing" containerID="cri-o://53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68" gracePeriod=25
Nov 01 22:23:44.140586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.140555    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" podUID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" containerName="payload-pre-processing" containerID="cri-o://b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c" gracePeriod=25
Nov 01 22:23:44.437911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.437886    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:23:44.477165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.477139    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:23:44.580872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.580843    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\") pod \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") "
Nov 01 22:23:44.581066 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.580981    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\" (UniqueName: \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\") pod \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") "
Nov 01 22:23:44.581288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.581253    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\") pod \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\" (UID: \"7daef227-30bb-4da9-b33a-6c0ae9606e0c\") "
Nov 01 22:23:44.581988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.581353    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\" (UniqueName: \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\") pod \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\" (UID: \"ae3ff83b-25ff-4150-9cf2-1329d4b664e1\") "
Nov 01 22:23:44.581988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.581453    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume" pod "ae3ff83b-25ff-4150-9cf2-1329d4b664e1" (UID: "ae3ff83b-25ff-4150-9cf2-1329d4b664e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:23:44.581988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.581655    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume" pod "7daef227-30bb-4da9-b33a-6c0ae9606e0c" (UID: "7daef227-30bb-4da9-b33a-6c0ae9606e0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:23:44.582143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.581997    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7daef227-30bb-4da9-b33a-6c0ae9606e0c-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:44.582143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.582019    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:44.583587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.583564    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c" pod "ae3ff83b-25ff-4150-9cf2-1329d4b664e1" (UID: "ae3ff83b-25ff-4150-9cf2-1329d4b664e1"). InnerVolumeSpecName "kube-api-access-6ls8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:44.583715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.583699    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp" pod "7daef227-30bb-4da9-b33a-6c0ae9606e0c" (UID: "7daef227-30bb-4da9-b33a-6c0ae9606e0c"). InnerVolumeSpecName "kube-api-access-qr8qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:44.684132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.684044    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ls8c\" (UniqueName: \"kubernetes.io/projected/ae3ff83b-25ff-4150-9cf2-1329d4b664e1-kube-api-access-6ls8c\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:44.684132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.684084    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8qp\" (UniqueName: \"kubernetes.io/projected/7daef227-30bb-4da9-b33a-6c0ae9606e0c-kube-api-access-qr8qp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:44.766517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.766481    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:23:44.781768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:44.781718    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:23:45.168074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.161039    5845 generic.go:356] "Generic (PLEG): container finished" podID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" containerID="b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c" exitCode=0
Nov 01 22:23:45.168074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.161217    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" event={"ID":"7daef227-30bb-4da9-b33a-6c0ae9606e0c","Type":"ContainerDied","Data":"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"}
Nov 01 22:23:45.168074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.161278    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq" event={"ID":"7daef227-30bb-4da9-b33a-6c0ae9606e0c","Type":"ContainerDied","Data":"a222e1a455799fcaaaa9585f0f7d59cecfa3f6b678c8663711d525c77f08a7f1"}
Nov 01 22:23:45.168074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.161326    5845 scope.go:122] "RemoveContainer" containerID="b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"
Nov 01 22:23:45.168074 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.161488    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"
Nov 01 22:23:45.172630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.171606    5845 generic.go:356] "Generic (PLEG): container finished" podID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" containerID="53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68" exitCode=0
Nov 01 22:23:45.172630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.171874    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" event={"ID":"ae3ff83b-25ff-4150-9cf2-1329d4b664e1","Type":"ContainerDied","Data":"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"}
Nov 01 22:23:45.172630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.171926    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk" event={"ID":"ae3ff83b-25ff-4150-9cf2-1329d4b664e1","Type":"ContainerDied","Data":"e719bdf10ac54c0c263f3f4128de111c9133e40002c5cdcd4b0ea18b5ff2adb6"}
Nov 01 22:23:45.172630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.172003    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"
Nov 01 22:23:45.208427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.208398    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:23:45.215372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.215347    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-6d84998bb-2gwsk"]
Nov 01 22:23:45.229680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.229626    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:23:45.233740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.233720    5845 scope.go:122] "RemoveContainer" containerID="b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"
Nov 01 22:23:45.234893 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:45.234487    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c\": container with ID starting with b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c not found: ID does not exist" containerID="b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"
Nov 01 22:23:45.234893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.234529    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c"} err="failed to get container status \"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c\": rpc error: code = NotFound desc = could not find container \"b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c\": container with ID starting with b0a85d5ec770154e711cde067d70e9e2f93a48d87758b44e0e57d5f0e1a3bd2c not found: ID does not exist"
Nov 01 22:23:45.234893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.234554    5845 scope.go:122] "RemoveContainer" containerID="53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"
Nov 01 22:23:45.239472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.239455    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e25317-dc6d48575-n4lpq"]
Nov 01 22:23:45.251261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.251233    5845 scope.go:122] "RemoveContainer" containerID="53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"
Nov 01 22:23:45.251684 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:45.251655    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68\": container with ID starting with 53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68 not found: ID does not exist" containerID="53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"
Nov 01 22:23:45.251759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:45.251693    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68"} err="failed to get container status \"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68\": rpc error: code = NotFound desc = could not find container \"53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68\": container with ID starting with 53d7b972f898a6a73a610a856f2c891902687697bda3b67b8fa96d64ef87da68 not found: ID does not exist"
Nov 01 22:23:46.869696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:46.869651    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" path="/var/lib/kubelet/pods/7daef227-30bb-4da9-b33a-6c0ae9606e0c/volumes"
Nov 01 22:23:46.870518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:46.870491    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" path="/var/lib/kubelet/pods/ae3ff83b-25ff-4150-9cf2-1329d4b664e1/volumes"
Nov 01 22:23:49.766968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:49.766924    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" podUID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" containerName="payload-processing" containerID="cri-o://90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088" gracePeriod=25
Nov 01 22:23:49.782191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:49.782142    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing" containerID="cri-o://913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb" gracePeriod=25
Nov 01 22:23:50.048062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.048037    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:23:50.089950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.089914    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\" (UniqueName: \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\") pod \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") "
Nov 01 22:23:50.090333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.090291    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\") pod \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\" (UID: \"ee2bdafc-355f-41b4-aa9c-11e32f374db3\") "
Nov 01 22:23:50.090658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.090631    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume" pod "ee2bdafc-355f-41b4-aa9c-11e32f374db3" (UID: "ee2bdafc-355f-41b4-aa9c-11e32f374db3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:23:50.091124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.091074    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee2bdafc-355f-41b4-aa9c-11e32f374db3-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:50.092340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.092310    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj" pod "ee2bdafc-355f-41b4-aa9c-11e32f374db3" (UID: "ee2bdafc-355f-41b4-aa9c-11e32f374db3"). InnerVolumeSpecName "kube-api-access-tz8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:23:50.155606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.155570    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:23:50.194243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.194187    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8sj\" (UniqueName: \"kubernetes.io/projected/ee2bdafc-355f-41b4-aa9c-11e32f374db3-kube-api-access-tz8sj\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:23:50.224330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.224281    5845 generic.go:356] "Generic (PLEG): container finished" podID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" containerID="90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088" exitCode=0
Nov 01 22:23:50.224504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.224356    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"
Nov 01 22:23:50.224504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.224418    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" event={"ID":"ee2bdafc-355f-41b4-aa9c-11e32f374db3","Type":"ContainerDied","Data":"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"}
Nov 01 22:23:50.224504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.224475    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf" event={"ID":"ee2bdafc-355f-41b4-aa9c-11e32f374db3","Type":"ContainerDied","Data":"67b4db864e9aa75fa8d485811913458241840b712166007b1289894d27d58be5"}
Nov 01 22:23:50.224504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.224499    5845 scope.go:122] "RemoveContainer" containerID="90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"
Nov 01 22:23:50.236082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.236066    5845 scope.go:122] "RemoveContainer" containerID="90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"
Nov 01 22:23:50.236427 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:23:50.236406    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088\": container with ID starting with 90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088 not found: ID does not exist" containerID="90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"
Nov 01 22:23:50.236483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.236435    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088"} err="failed to get container status \"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088\": rpc error: code = NotFound desc = could not find container \"90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088\": container with ID starting with 90e39a51077dbbcd180352b2da8963ab98e734d4ea95fdd11fa7eae957a37088 not found: ID does not exist"
Nov 01 22:23:50.249659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.249637    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:23:50.254842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.254823    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b4c49-64cdb676c4-pq9nf"]
Nov 01 22:23:50.863118 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:50.863086    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" path="/var/lib/kubelet/pods/ee2bdafc-355f-41b4-aa9c-11e32f374db3/volumes"
Nov 01 22:23:52.770977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.770945    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:23:52.773704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.773664    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" containerName="payload-processing"
Nov 01 22:23:52.773898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.773884    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" containerName="payload-processing"
Nov 01 22:23:52.774063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774051    5845 cpu_manager.go:414] "containerMap: removing container" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy"
Nov 01 22:23:52.774167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774155    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy"
Nov 01 22:23:52.774274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774263    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" containerName="payload-pre-processing"
Nov 01 22:23:52.774406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774393    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" containerName="payload-pre-processing"
Nov 01 22:23:52.774599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774588    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" containerName="payload-processing"
Nov 01 22:23:52.774678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774667    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" containerName="payload-processing"
Nov 01 22:23:52.774763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774753    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.774832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774823    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.774934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774924    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.775006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.774997    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.775417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775400    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="59b5e520-f8b0-41b6-91a9-7b030654b9e4" containerName="istio-proxy"
Nov 01 22:23:52.775546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775535    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7daef227-30bb-4da9-b33a-6c0ae9606e0c" containerName="payload-pre-processing"
Nov 01 22:23:52.775663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775652    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3ff83b-25ff-4150-9cf2-1329d4b664e1" containerName="payload-processing"
Nov 01 22:23:52.775745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775736    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.775829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775818    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f10c1d2-c233-4174-939b-676ec58c594d" containerName="istio-proxy"
Nov 01 22:23:52.775915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.775906    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2bdafc-355f-41b4-aa9c-11e32f374db3" containerName="payload-processing"
Nov 01 22:23:52.781278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.781249    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:23:52.781431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.781415    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.785541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.785518    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ad047b-openshift-default-dockercfg-jt46b\""
Nov 01 22:23:52.845562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845528    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.845720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845595    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.845720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845709    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845761    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845878    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845901    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfvw\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.845921    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.846033    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.846386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.846061    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947234    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947366    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947517    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947553    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfvw\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947584    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947697    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947761    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947818    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.947883    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.949346    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.949867    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.950420    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.949332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.950597    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.952820 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.950855    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.952820 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.952014    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.953122 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.953100    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.961589 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.961568    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:52.962087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:52.962061    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfvw\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\") pod \"e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") " pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:53.090542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.090463    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:23:53.097645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.097620    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.100133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.100109    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:53.112141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.111803    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:23:53.150450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.150416    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.150679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.150654    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68zr\" (UniqueName: \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.200641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.185753    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e25317\""
Nov 01 22:23:53.200641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.185937    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e25317-dockercfg-ftwmj\""
Nov 01 22:23:53.278321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.277381    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68zr\" (UniqueName: \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.278321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.277689    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.349563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.327204    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.349563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.332425    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68zr\" (UniqueName: \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\") pod \"payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.423423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.422867    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:53.606552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.606401    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:23:53.610896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.610867    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:23:53.610974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.610932    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:23:53.610974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.610962    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:23:53.721322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.721276    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:23:53.721541 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:23:53.721514    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb26c1d_9648_4313_b54c_14c616894e4e.slice/crio-68a7ab637453d98db28240f5156a1b512617f8fea9d31d397bb46c40a0088208 WatchSource:0}: Error finding container 68a7ab637453d98db28240f5156a1b512617f8fea9d31d397bb46c40a0088208: Status 404 returned error can't find the container with id 68a7ab637453d98db28240f5156a1b512617f8fea9d31d397bb46c40a0088208
Nov 01 22:23:53.724791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:53.724773    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:23:54.376479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:54.376431    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerStarted","Data":"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"}
Nov 01 22:23:54.376934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:54.376496    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerStarted","Data":"47a0bdf6d793463c64bfa3cc43aa825bbef767ab55162966d0174d7a406f325b"}
Nov 01 22:23:54.378385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:54.378349    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" event={"ID":"4eb26c1d-9648-4313-b54c-14c616894e4e","Type":"ContainerStarted","Data":"68a7ab637453d98db28240f5156a1b512617f8fea9d31d397bb46c40a0088208"}
Nov 01 22:23:55.111038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:55.105658    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:55.133963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:55.133927    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:55.166735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:55.166647    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podStartSLOduration=3.166628313 podStartE2EDuration="3.166628313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:54.406915714 +0000 UTC m=+2337.663095478" watchObservedRunningTime="2026-11-01 22:23:55.166628313 +0000 UTC m=+2338.422808077"
Nov 01 22:23:55.463373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:55.456760    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:55.505964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:55.493728    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:23:56.459185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:56.459093    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" event={"ID":"4eb26c1d-9648-4313-b54c-14c616894e4e","Type":"ContainerStarted","Data":"a9b0512973b3bf47374301533afbaa05b0fcb70615116ffe534c81aac5696b79"}
Nov 01 22:23:56.459360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:56.459186    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:23:56.482555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:23:56.482495    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" podStartSLOduration=1.206531085 podStartE2EDuration="3.482482749s" totalImagesPullingTime="2.275951664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:56.479022347 +0000 UTC m=+2339.735202114" watchObservedRunningTime="2026-11-01 22:23:56.482482749 +0000 UTC m=+2339.738662512"
Nov 01 22:24:00.155479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:00.155385    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:24:04.697325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.696856    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:24:04.704854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.703539    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.715176 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.715149    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-09656e-openshift-default-dockercfg-j7hg2\""
Nov 01 22:24:04.722358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.722325    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:24:04.723139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723115    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723162    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723376    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723428    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723499 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723481    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xll\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723587    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723632    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723716 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723702    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.723750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.723732    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.863677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.863621    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.958512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.958422    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.958947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.958922    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.959091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.959075    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.959259 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.959244    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.959435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.959419    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.985341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.959689    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.987681 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.987654    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.987911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.987890    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xll\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:04.989691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.887075    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.007388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.989810    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.031693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.987086    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.032012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:04.987315    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.032521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.006690    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.043440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.043383    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.044051 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.043997    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.061348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.048643    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xll\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.102635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.102386    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\") pod \"e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") " pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.110082 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:24:05.110049    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:24:05.330745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.330713    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:05.676817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.676788    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:24:05.677062 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:05.677043    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb204b3aa_d4fe_48b1_93b7_888db9d93a20.slice/crio-dcf2b19cb28f4fc578470ceae013787462385f34e87eefaad95b0bca47d88fba WatchSource:0}: Error finding container dcf2b19cb28f4fc578470ceae013787462385f34e87eefaad95b0bca47d88fba: Status 404 returned error can't find the container with id dcf2b19cb28f4fc578470ceae013787462385f34e87eefaad95b0bca47d88fba
Nov 01 22:24:05.680517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.680487    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:05.680616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.680550    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:05.680616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:05.680577    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:06.689124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:06.689089    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerStarted","Data":"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"}
Nov 01 22:24:06.689645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:06.689140    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerStarted","Data":"dcf2b19cb28f4fc578470ceae013787462385f34e87eefaad95b0bca47d88fba"}
Nov 01 22:24:06.717562 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:06.717503    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podStartSLOduration=2.717485119 podStartE2EDuration="2.717485119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:06.713430228 +0000 UTC m=+2349.969610006" watchObservedRunningTime="2026-11-01 22:24:06.717485119 +0000 UTC m=+2349.973664883"
Nov 01 22:24:07.331759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:07.331724    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:07.337854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:07.337830    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:07.513694 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:07.513653    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:24:07.700525 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:07.700438    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:07.701685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:07.701661    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:24:09.351424 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:09.351385    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:24:09.352022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:09.351444    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:24:10.248112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.248075    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:24:10.337191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.337152    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\" (UniqueName: \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\") pod \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") "
Nov 01 22:24:10.337377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.337248    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\") pod \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\" (UID: \"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2\") "
Nov 01 22:24:10.337616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.337591    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume" pod "2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" (UID: "2f063e31-bd21-40e7-a65e-ebe5e2fdafb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:10.337904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.337887    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:10.376431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.376398    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws" pod "2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" (UID: "2f063e31-bd21-40e7-a65e-ebe5e2fdafb2"). InnerVolumeSpecName "kube-api-access-hnxws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:10.440411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.440321    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnxws\" (UniqueName: \"kubernetes.io/projected/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2-kube-api-access-hnxws\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:10.733852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.733811    5845 generic.go:356] "Generic (PLEG): container finished" podID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerID="913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb" exitCode=0
Nov 01 22:24:10.734033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.734016    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" event={"ID":"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2","Type":"ContainerDied","Data":"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"}
Nov 01 22:24:10.734091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.734058    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" event={"ID":"2f063e31-bd21-40e7-a65e-ebe5e2fdafb2","Type":"ContainerDied","Data":"b0e671f1b74301a8152813f9ef2a63a647b0fb449877c0c1c9493b624de343e6"}
Nov 01 22:24:10.734091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.734082    5845 scope.go:122] "RemoveContainer" containerID="913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"
Nov 01 22:24:10.734260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.734245    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"
Nov 01 22:24:10.775284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.775256    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:24:10.780965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.780940    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l"]
Nov 01 22:24:10.802167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.802147    5845 scope.go:122] "RemoveContainer" containerID="913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"
Nov 01 22:24:10.802528 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:24:10.802507    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb\": container with ID starting with 913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb not found: ID does not exist" containerID="913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"
Nov 01 22:24:10.802573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.802539    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb"} err="failed to get container status \"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb\": rpc error: code = NotFound desc = could not find container \"913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb\": container with ID starting with 913884aabe7ab5a4637ff38d01639712f889a4e0f664e5db84e3ee5d7d3e5eeb not found: ID does not exist"
Nov 01 22:24:10.865087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:10.865057    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" path="/var/lib/kubelet/pods/2f063e31-bd21-40e7-a65e-ebe5e2fdafb2/volumes"
Nov 01 22:24:11.187378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:11.183470    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6b4c49-5cdb7cdc9f-7448l" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.147:50052\" within 1s: context deadline exceeded"
Nov 01 22:24:11.287637 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:11.287604    5845 logging.go:55] [core] [Channel #6621 SubChannel #6622]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.147:50052", ServerName: "10.134.0.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.147:50052: operation was canceled"
Nov 01 22:24:16.090144 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.090110    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:24:16.090950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.090932    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing"
Nov 01 22:24:16.090999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.090951    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing"
Nov 01 22:24:16.091069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.091060    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f063e31-bd21-40e7-a65e-ebe5e2fdafb2" containerName="payload-pre-processing"
Nov 01 22:24:16.288316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.281601    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:24:16.288316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.281775    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.329401 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.325744    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-829cb0-dockercfg-56q77\""
Nov 01 22:24:16.329401 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.325851    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-829cb0\""
Nov 01 22:24:16.379463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.365779    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8g4\" (UniqueName: \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.379463 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.376191    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.477763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.477726    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8g4\" (UniqueName: \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.477930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.477891    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.478561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.478541    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.490169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.490138    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8g4\" (UniqueName: \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\") pod \"payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:16.706206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:16.706130    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:17.233719 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:17.228531    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:24:17.233719 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:17.229718    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d07a690_ca3c_4d37_941d_68b7ef26f366.slice/crio-abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570 WatchSource:0}: Error finding container abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570: Status 404 returned error can't find the container with id abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570
Nov 01 22:24:17.950173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:17.950131    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" event={"ID":"6d07a690-ca3c-4d37-941d-68b7ef26f366","Type":"ContainerStarted","Data":"e299590a7f7aa3056c59f43cebaa1fe18d25d709a8c45b706097fc488774c89c"}
Nov 01 22:24:17.950512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:17.950197    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" event={"ID":"6d07a690-ca3c-4d37-941d-68b7ef26f366","Type":"ContainerStarted","Data":"abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570"}
Nov 01 22:24:17.950512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:17.950466    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:17.971905 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:17.971853    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podStartSLOduration=1.971839824 podStartE2EDuration="1.971839824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:17.969347305 +0000 UTC m=+2361.225527069" watchObservedRunningTime="2026-11-01 22:24:17.971839824 +0000 UTC m=+2361.228019600"
Nov 01 22:24:21.993212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.993176    5845 generic.go:356] "Generic (PLEG): container finished" podID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerID="bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092" exitCode=0
Nov 01 22:24:21.993670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.993226    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerDied","Data":"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"}
Nov 01 22:24:21.993760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.993743    5845 scope.go:122] "RemoveContainer" containerID="bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"
Nov 01 22:24:21.995680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.995650    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:21.995771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.995725    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:21.995771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:21.995758    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:23.006073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:23.006039    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerStarted","Data":"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"}
Nov 01 22:24:23.308702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:23.308625    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:24:23.832670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:23.832634    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:24:23.833224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:23.833197    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 22:24:23.833367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:23.833251    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 22:24:24.832698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:24.832654    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 22:24:24.833622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:24.832723    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 22:24:25.835325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:25.832242    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 22:24:25.835325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:25.832335    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 22:24:26.684344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.684283    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:24:26.690417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.690388    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:24:26.691099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.690492    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.695089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.695063    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0bb375-openshift-default-dockercfg-p4x9v\""
Nov 01 22:24:26.732275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732244    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732451 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732411    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732531 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732471    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm75l\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732583    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732626    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732748    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732793    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732899    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.732973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.732929    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.836326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.836197    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 22:24:26.836326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.836259    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 22:24:26.837757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.837726    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.838028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.838008    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.841403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.841377    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm75l\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.841587 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.841572    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.857379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.841689    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.861480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.861452    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.867420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.867393    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.867678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.867659    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.867805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.867789    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.868745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.868144    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.868892 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.860065    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.876171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.876097    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.876171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.866218    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.881311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.877446    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.894884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.894857    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.934333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.930038    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm75l\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.934333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.932282    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:26.948378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:26.945055    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\") pod \"e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") " pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:27.011800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.011749    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:27.363171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.362954    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:24:27.367080 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:27.363822    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637a4546_919b_448a_80da_944af0aaf4b3.slice/crio-369262d1f115b28765baa46c162ef212c4269b25a48a9db48a781efb48ebd108 WatchSource:0}: Error finding container 369262d1f115b28765baa46c162ef212c4269b25a48a9db48a781efb48ebd108: Status 404 returned error can't find the container with id 369262d1f115b28765baa46c162ef212c4269b25a48a9db48a781efb48ebd108
Nov 01 22:24:27.374386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.369065    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:27.374386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.369157    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:27.374386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.369206    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:24:27.833450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.833100    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body=
Nov 01 22:24:27.833450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:27.833157    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused"
Nov 01 22:24:28.079853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.079819    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" event={"ID":"637a4546-919b-448a-80da-944af0aaf4b3","Type":"ContainerStarted","Data":"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"}
Nov 01 22:24:28.080430 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.080402    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" event={"ID":"637a4546-919b-448a-80da-944af0aaf4b3","Type":"ContainerStarted","Data":"369262d1f115b28765baa46c162ef212c4269b25a48a9db48a781efb48ebd108"}
Nov 01 22:24:28.117499 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.117400    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" podStartSLOduration=2.117385868 podStartE2EDuration="2.117385868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:28.114834562 +0000 UTC m=+2371.371014338" watchObservedRunningTime="2026-11-01 22:24:28.117385868 +0000 UTC m=+2371.373565632"
Nov 01 22:24:28.885545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.885511    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:24:28.887237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.887213    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:24:28.975406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:28.975373    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:24:29.012776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:29.012748    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:29.018821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:29.018795    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:29.113545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:29.113505    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:29.115087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:29.115058    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:24:31.573446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.573410    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:24:31.578030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.578008    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.589926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.589894    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:24:31.592510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.592485    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0bb375\""
Nov 01 22:24:31.615700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.615671    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:24:31.627522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.627498    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:24:31.627687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.627616    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.634085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.634061    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0bb375-dockercfg-9d2x2\""
Nov 01 22:24:31.671749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.671712    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8cb\" (UniqueName: \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.671967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.671884    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.672044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.671974    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqtj\" (UniqueName: \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.672161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.672136    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.787376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.783265    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.787376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.783486    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8cb\" (UniqueName: \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.787376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.783618    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.787376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.783705    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqtj\" (UniqueName: \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.787376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.784972    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.808318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.799253    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.820158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.814891    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqtj\" (UniqueName: \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\") pod \"payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:31.841377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.840820    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8cb\" (UniqueName: \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\") pod \"payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.920182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.918681    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:31.947916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:31.946695    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:32.256361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:32.256331    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:24:32.256946 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:32.256916    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f7df48_8487_496a_a5c5_743eb09f9fa2.slice/crio-47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7 WatchSource:0}: Error finding container 47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7: Status 404 returned error can't find the container with id 47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7
Nov 01 22:24:32.258826 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:24:32.258799    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba174d90_0313_497b_a8e8_7d037efd859d.slice/crio-2c5a8c2c53a902a94653258350ac7ca06a5220a828f49e9d2adcaaee11c173aa WatchSource:0}: Error finding container 2c5a8c2c53a902a94653258350ac7ca06a5220a828f49e9d2adcaaee11c173aa: Status 404 returned error can't find the container with id 2c5a8c2c53a902a94653258350ac7ca06a5220a828f49e9d2adcaaee11c173aa
Nov 01 22:24:32.260782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:32.260759    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:24:33.146541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.146498    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" event={"ID":"a8f7df48-8487-496a-a5c5-743eb09f9fa2","Type":"ContainerStarted","Data":"1aa19a93aea78a20c14527807ea35d805bf9e150d870bfb3cf6cbc1e53d0609f"}
Nov 01 22:24:33.146990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.146567    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" event={"ID":"a8f7df48-8487-496a-a5c5-743eb09f9fa2","Type":"ContainerStarted","Data":"47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7"}
Nov 01 22:24:33.146990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.146600    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:33.148275 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.148252    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" event={"ID":"ba174d90-0313-497b-a8e8-7d037efd859d","Type":"ContainerStarted","Data":"c0f561df56a980cf2a6722ebce9c6c8a34c6ff42cee89297f47e6e6d1bf3649a"}
Nov 01 22:24:33.148378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.148322    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" event={"ID":"ba174d90-0313-497b-a8e8-7d037efd859d","Type":"ContainerStarted","Data":"2c5a8c2c53a902a94653258350ac7ca06a5220a828f49e9d2adcaaee11c173aa"}
Nov 01 22:24:33.148378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.148344    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:33.169760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.169711    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" podStartSLOduration=2.169693954 podStartE2EDuration="2.169693954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:33.167457136 +0000 UTC m=+2376.423636900" watchObservedRunningTime="2026-11-01 22:24:33.169693954 +0000 UTC m=+2376.425873721"
Nov 01 22:24:33.190858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:33.190811    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" podStartSLOduration=2.190799513 podStartE2EDuration="2.190799513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:33.185453804 +0000 UTC m=+2376.441633578" watchObservedRunningTime="2026-11-01 22:24:33.190799513 +0000 UTC m=+2376.446979318"
Nov 01 22:24:34.822236 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:24:34.822191    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:24:44.159573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:44.159538    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:24:44.159985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:44.159613    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:24:59.221347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:59.210755    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:24:59.232395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:24:59.232373    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:25:00.112926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.111515    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:25:00.124672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.123074    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:25:00.124672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.123194    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.131253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.130134    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad047b\""
Nov 01 22:25:00.131253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.130328    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad047b-dockercfg-v89dc\""
Nov 01 22:25:00.369419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.367811    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.369419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.368004    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpzzl\" (UniqueName: \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.407357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.398165    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:25:00.519323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.476554    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.519323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.476734    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpzzl\" (UniqueName: \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.519323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.494112    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.558330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.555007    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpzzl\" (UniqueName: \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\") pod \"payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:00.843321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:00.843183    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:01.413793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:01.413753    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:25:01.415881 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:01.415848    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372ed1e0_298d_49c3_8b50_61c0116b3334.slice/crio-e88a335e621a17b744f7afc0b5a57e8127867bd0872a15c804a21df17a70d58c WatchSource:0}: Error finding container e88a335e621a17b744f7afc0b5a57e8127867bd0872a15c804a21df17a70d58c: Status 404 returned error can't find the container with id e88a335e621a17b744f7afc0b5a57e8127867bd0872a15c804a21df17a70d58c
Nov 01 22:25:01.785482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:01.785390    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" event={"ID":"372ed1e0-298d-49c3-8b50-61c0116b3334","Type":"ContainerStarted","Data":"e88a335e621a17b744f7afc0b5a57e8127867bd0872a15c804a21df17a70d58c"}
Nov 01 22:25:02.812247 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:02.812202    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" event={"ID":"372ed1e0-298d-49c3-8b50-61c0116b3334","Type":"ContainerStarted","Data":"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"}
Nov 01 22:25:02.812730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:02.812461    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:25:02.835517 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:02.835470    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" podStartSLOduration=2.835457737 podStartE2EDuration="2.835457737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:02.83200756 +0000 UTC m=+2406.088187323" watchObservedRunningTime="2026-11-01 22:25:02.835457737 +0000 UTC m=+2406.091637500"
Nov 01 22:25:04.839866 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:25:04.839821    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:25:05.469564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:05.469511    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing" containerID="cri-o://e299590a7f7aa3056c59f43cebaa1fe18d25d709a8c45b706097fc488774c89c" gracePeriod=25
Nov 01 22:25:05.904283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:05.904244    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:25:05.907401 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:05.907338    5845 generic.go:356] "Generic (PLEG): container finished" podID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerID="e299590a7f7aa3056c59f43cebaa1fe18d25d709a8c45b706097fc488774c89c" exitCode=0
Nov 01 22:25:06.035512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.035475    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" event={"ID":"6d07a690-ca3c-4d37-941d-68b7ef26f366","Type":"ContainerDied","Data":"e299590a7f7aa3056c59f43cebaa1fe18d25d709a8c45b706097fc488774c89c"}
Nov 01 22:25:06.035684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.035515    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.035684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.035535    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:25:06.108614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108589    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.108758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108745    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108789    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108806    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ggp\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108923    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108953    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.108979    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.109020    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.111506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.109146    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.210976    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211092    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211259 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211241    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211346 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211323    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211445    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211523 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211488    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ggp\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211538    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211571    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.211622 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211606    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.231748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.211652    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.277048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.233941    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.277048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.234029    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.277048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.234048    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.277048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.234076    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:06.955273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.955239    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:07.014690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.955291    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:07.014690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.955340    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:07.014690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:06.955541    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ggp\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\") pod \"e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") " pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:07.124936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:07.124903    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:25:10.936065 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:10.936023    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.179:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.169203    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.177656    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.177706    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.179948    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:10.702401    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.177626    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:13.177735    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:14.005979    5845 logging.go:55] [core] [Channel #6869 SubChannel #6870]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.179:50052", ServerName: "10.134.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.179:50052: connect: connection refused"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.287805    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.657148    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.657214    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.659081    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.158:8443/health\": context deadline exceeded"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.660751    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:14.839045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:14.806706    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:15.922829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:15.327659    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:15.922829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:15.594281    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:15.922829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:15.596482    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:26.127153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:25.922662    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:26.616604    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:26.883842    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:27.771084    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:27.864476    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.083071    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.083165    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.095250    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.415931    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.463256    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.475641    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.533597    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.80:50052: i/o timeout"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.563935    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.610635    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.87:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.610815    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.716758    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.770780    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.786464    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.99:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.788191    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.788270    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": net/http: TLS handshake timeout"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.788467    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.872378    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6db8848b-ac26-44f3-855f-a5217ee57d81" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.920312    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.920367    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.920614    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.920642    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:28.981267    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": EOF"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.029686    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.313525564s: [/var/lib/containers/storage/overlay/c0b1f294ce4046fa3be2af91891c31d4f05fcc55490927d236ae0be74e29dde8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.030233    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.031106    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.315577224s: [/var/lib/containers/storage/overlay/9d28647f1faad4a326e79728aced19629f1fca673a7335141a4311a557383d86/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h_f123a7c8-5b58-4630-8eb1-4984e30e9cfb/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.031129    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.031181    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.031769    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.3388224s: [/var/lib/containers/storage/overlay/f7818d7baaddb0aaa1abcdc9c95feca888a6a1a0c6bb87e71e1cd95aebf03a48/diff /var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm_0a2d2561-28ba-4f2e-bd41-88b863f73b11/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:25.760312    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.043741    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.062273    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.063689    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.065009    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.302551599s: [/var/lib/containers/storage/overlay/801b7678fd56705e4d8ee49172d4830949015ca193b772efa1c7268ef6f08948/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.065769    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.302958506s: [/var/lib/containers/storage/overlay/07adbe6aaa91f03a8d20c20420a2a7e6a2adc358f2a86979bb888ccd1663f31a/diff /var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-kdssh_ed7e30f6-adec-4f93-a098-2ab4f61f937d/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.066704    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:25.732541    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.102:8443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:25.732565    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.069569    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.073883    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.82:8443/health\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.076645    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.157:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.081839    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.160:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.128797    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6db8848b-ac26-44f3-855f-a5217ee57d81" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.128820    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.128864    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.129193    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.129216    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.130726    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.507320217s: [/var/lib/containers/storage/overlay/b7bc7d6ccfe8d7106cc0d91ddf443f6243b8bde5381984b9b5d3345f35ea02f2/diff /var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m_83f9f654-a050-4cdc-bc76-d2ea43698bdd/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.133597    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.419051916s: [/var/lib/containers/storage/overlay/e8eceb5fd600399544ec22ce481f853085309b59d9c4a24ece1818826596943e/diff /var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cdk5r_b27c16ff-fa1f-4815-902c-8a05f5070f07/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.134207    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.420448512s: [/var/lib/containers/storage/overlay/866dedf9560157e9de21c250c282cab7a791e12fe7207c5a7ae89f36b7408e0f/diff /var/log/pods/openshift-ingress_e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9_ae7bbaad-d051-4782-a918-386d7f69bd6d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.135858    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.135906    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.139391    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.177:9004: i/o timeout"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.140963    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.143331    5845 logging.go:55] [core] [Channel #6877 SubChannel #6892]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.157:50052: operation was canceled"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.254522    5845 logging.go:55] [core] [Channel #6875 SubChannel #6882]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.160:50052: operation was canceled"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.261558    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.262468    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.262511    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.263814    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.263923    5845 logging.go:55] [core] [Channel #6871 SubChannel #6872]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.99:50052", ServerName: "10.134.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:55432->10.134.0.99:50052: use of closed network connection"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.264394    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.264433    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.264490    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.502435    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.570668    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571354    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571415    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571459    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571529    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571653    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.571712    5845 logging.go:55] [core] [Channel #6880 SubChannel #6893]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.156:50052: operation was canceled"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.571732    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.572097    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.572185    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.572208    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.572586    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.573273    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.573331    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.579244    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.342488    5845 logging.go:55] [core] [Channel #6878 SubChannel #6889]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.182:50052: operation was canceled"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.594436    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.596938    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.85:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.611232    5845 trace.go:236] Trace[386890577]: "Calculate volume metrics of konnectivity-ca for pod kube-system/konnectivity-agent-9wtgk" (01-Nov-2026 22:25:28.609) (total time: 1001ms):
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: Trace[386890577]: [1.001321707s] [1.001321707s] END
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:25:29.623449    5845 logging.go:55] [core] [Channel #6909 SubChannel #6933]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.179:50052", ServerName: "10.134.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.179:50052: connect: connection refused"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.644493    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.759522    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.767114    5845 trace.go:236] Trace[571533689]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" (01-Nov-2026 22:25:28.594) (total time: 1172ms):
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: Trace[571533689]: [1.172142057s] [1.172142057s] END
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.810767    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.810827    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.818253    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.920696    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.920774    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:29.977281    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.025486    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.025528    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.025639    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.031890    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.031929    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.044219    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.044270    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.104380    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.104423    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.106511    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.106553    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.108389    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.243204    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.042553257s: [/var/lib/containers/storage/overlay/37af5a00d0bc15ea2add4a7f59d709094a58e326392c263709bb0464d9e322c6/diff /var/log/pods/openshift-ingress_e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz_a1e13660-8961-418b-afd7-1bdf2ee53d34/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.255934    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.255980    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.265546    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.265584    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.289378    5845 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dsdmp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.289409    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.303424    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.303458    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.305377    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.365487    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.815438457s: [/var/lib/containers/storage/overlay/82171a2307ee847a996883e9b7815b3a7771d342f655594700eb8c71dbb57fc3/diff /var/log/pods/openshift-ingress_e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq_637a4546-919b-448a-80da-944af0aaf4b3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.586845    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.12:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.586893    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.588660    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.179:50052\" within 1s: context deadline exceeded"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.615664    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.615706    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.617393    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.656406    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.656444    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:30.691754    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.43:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:31.083933    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.29:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:31.642752    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:31.642820    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:31.698455    5845 patch_prober.go:29] interesting pod/jobset-controller-manager-75d8948445-6nwm2 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:25:37.787160 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:31.698537    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2" podUID="533a4e94-3fb6-4bb4-b248-0a256638d509" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:32.303393    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:32.303443    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:32.656393    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:32.656455    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:33.445047    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:33.445106    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:33.711570    5845 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-kwlgq container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:33.711628    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq" podUID="cefe7ab7-6f8a-4895-af7f-747da1036866" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:00.207255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:25:34.108417    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.615832    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.615947    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.631027    5845 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dsdmp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.631096    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.654418    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.702489    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-tbrm6" podUID="726e6b52-8dbd-4f88-9535-77748f546abb" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.23:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.808877    5845 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.712239464s: [/var/lib/containers/storage/overlay/c5421b6de96e61e93dbe92e0cf974849529972d7a98a5026de848cd848fc72ab/diff /var/log/pods/openshift-ingress_e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb_7366d1dd-7a69-47c6-983f-fb72ddccce54/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:06.839146    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.124352    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.124427    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.125405    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.127326    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.137180    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/health\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.137286    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.137715    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.80:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.152331    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.167:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.170801    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.106:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.178457    5845 logging.go:55] [core] [Channel #6973 SubChannel #6996]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.80:50052", ServerName: "10.134.0.80:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:40156->10.134.0.80:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.192438    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.193425    5845 logging.go:55] [core] [Channel #6974 SubChannel #6993]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.167:50052", ServerName: "10.134.0.167:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:43862->10.134.0.167:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.211462    5845 logging.go:55] [core] [Channel #6975 SubChannel #6986]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.106:50052", ServerName: "10.134.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52518->10.134.0.106:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.237448    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.237479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.387078    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.157:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.387070    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.86:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.387638    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.387677    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.393223    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.393656    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.393747    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.393834    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.427441    5845 logging.go:55] [core] [Channel #6971 SubChannel #6992]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49450->10.134.0.157:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.427466    5845 logging.go:55] [core] [Channel #6972 SubChannel #6978]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.86:50052", ServerName: "10.134.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59680->10.134.0.86:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.428385    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.431411    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.610028    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.610026    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.611583    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.611599    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.611625    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.611603    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.613131    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.617147    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.168:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.617763    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.617812    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.624273    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.99:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.627168    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.627328    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.627358    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.628757    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.160:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.632877    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.632915    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.633083    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.633118    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.636476    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.641823    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.643927    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.643965    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.644001    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.645472    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.105:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.646892    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.646909    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.646926    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.647719    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.647763    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.657477    5845 logging.go:55] [core] [Channel #6976 SubChannel #6983]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.168:50052", ServerName: "10.134.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49940->10.134.0.168:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.657805    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.660154    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.660184    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.664385    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.664418    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.665437    5845 logging.go:55] [core] [Channel #6977 SubChannel #6984]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.99:50052", ServerName: "10.134.0.99:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:35484->10.134.0.99:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.667422    5845 logging.go:55] [core] [Channel #6979 SubChannel #6988]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:38772->10.134.0.182:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.667599    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.667636    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.667833    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.667866    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.669440    5845 logging.go:55] [core] [Channel #6980 SubChannel #7007]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:32786->10.134.0.160:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.673384    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.675699    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.83:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.676926    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.676960    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.678858    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.681010    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.681205    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.681240    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.681247    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.85:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.682285    5845 patch_prober.go:29] interesting pod/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.682360    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.684412    5845 logging.go:55] [core] [Channel #6981 SubChannel #7008]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54536->10.134.0.156:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.685527    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.685570    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.686402    5845 logging.go:55] [core] [Channel #6982 SubChannel #6985]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49824->10.134.0.105:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.687402    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.687436    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.691549    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.12:9091/-/healthy\": context deadline exceeded" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.691580    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/healthy\": context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.691601    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.181:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.691666    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.691689    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.697459    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.697486    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.698407    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.698444    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.704068    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.706482    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.716035    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.716251    5845 patch_prober.go:29] interesting pod/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.716282    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.716432    5845 logging.go:55] [core] [Channel #6987 SubChannel #6990]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.83:50052", ServerName: "10.134.0.83:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:35706->10.134.0.83:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.717193    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.179:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.717284    5845 logging.go:55] [core] [Channel #7005 SubChannel #7009]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.179:50052", ServerName: "10.134.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.179:50052: operation was canceled"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.720384    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.720931    5845 patch_prober.go:29] interesting pod/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.720976    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.722414    5845 logging.go:55] [core] [Channel #6989 SubChannel #6991]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.85:50052", ServerName: "10.134.0.85:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49868->10.134.0.85:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:07.732426    5845 logging.go:55] [core] [Channel #6994 SubChannel #7001]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.181:50052", ServerName: "10.134.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:47842->10.134.0.181:50052: use of closed network connection"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:07.840125    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.031530    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6db8848b-ac26-44f3-855f-a5217ee57d81" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.031546    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6db8848b-ac26-44f3-855f-a5217ee57d81" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.031682    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-zw7wh" podUID="2ed0239d-76c2-4d3b-b168-560c022e4e3a" containerName="ovnkube-controller" probeResult="failure" output="command timed out"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.125587    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.125612    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.125587    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.128174    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.137776    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.137905    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.80:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.153343    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.167:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:13.307706 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.178459    5845 logging.go:55] [core] [Channel #7015 SubChannel #7016]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.80:50052", ServerName: "10.134.0.80:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:40166->10.134.0.80:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.194483    5845 logging.go:55] [core] [Channel #7018 SubChannel #7019]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.167:50052", ServerName: "10.134.0.167:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:43866->10.134.0.167:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.234435    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.387470    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.86:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.387595    5845 logging.go:55] [core] [Channel #7021 SubChannel #7022]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.86:50052", ServerName: "10.134.0.86:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59690->10.134.0.86:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.393337    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.393759    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.393831    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.393917    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.429370    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" podUID="60df8928-8bda-496d-a064-429629ed2583" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.610261    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.610311    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.611942    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.611959    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.611989    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.612232    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.617818    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.168:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.617933    5845 logging.go:55] [core] [Channel #7024 SubChannel #7025]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.168:50052", ServerName: "10.134.0.168:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49960->10.134.0.168:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.628352    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.628421    5845 logging.go:55] [core] [Channel #7027 SubChannel #7028]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:38776->10.134.0.182:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.633885    5845 patch_prober.go:29] interesting pod/thanos-querier-fb8c8bd9c-6mv4x container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.633934    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-fb8c8bd9c-6mv4x" podUID="f109cd89-d922-4952-b092-0360bbbbaf33" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.12:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.645054    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.645117    5845 logging.go:55] [core] [Channel #7030 SubChannel #7031]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54544->10.134.0.156:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.668615    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-mrlcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.668662    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-mrlcb" podUID="73a30e41-ed57-4bff-92a2-38160219bfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.692142    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.181:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:08.692201    5845 logging.go:55] [core] [Channel #7033 SubChannel #7034]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.181:50052", ServerName: "10.134.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:47856->10.134.0.181:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.700724    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.700763    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.704643    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.706836    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.840443    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.842433    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.87:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.977269    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:08.978647    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r" podUID="b27c16ff-fa1f-4815-902c-8a05f5070f07" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.019416    5845 patch_prober.go:29] interesting pod/dns-default-9jp45 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.019451    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-9jp45" podUID="55adb8b3-868b-4fe9-a2ee-a2de94739289" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.8:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.126161    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.128607    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.157133    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.177939    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.397968    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.398161    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.398404    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.398968    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.158:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.399445    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.29:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.399448    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.43:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.529599    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.105:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.529840    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.106:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:09.583104    5845 logging.go:55] [core] [Channel #7036 SubChannel #7038]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49832->10.134.0.105:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:09.583221    5845 logging.go:55] [core] [Channel #7037 SubChannel #7039]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.106:50052", ServerName: "10.134.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52528->10.134.0.106:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.622394    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.622777    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.624133    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.624423    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.631203    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.82:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.632102    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.632415    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.157:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.641885    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.649266    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.673507    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.673571    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:09.673682    5845 logging.go:55] [core] [Channel #7042 SubChannel #7043]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.157:50052", ServerName: "10.134.0.157:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49456->10.134.0.157:50052: use of closed network connection"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.673528    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.673737    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.698463    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.707987    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.102:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.730430    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.841327    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.970458    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.179:50052\" within 1s: context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:09.970537    5845 logging.go:55] [core] [Channel #7045 SubChannel #7046]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.179:50052", ServerName: "10.134.0.179:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.179:50052: operation was canceled"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:09.972503    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.127352    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.128829    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.401382    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.401962    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.402363    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.623139    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.623138    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.624721    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.624740    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.633091    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.689908    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:10.841698    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.127983    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm" podUID="0a2d2561-28ba-4f2e-bd41-88b863f73b11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.129384    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp" podUID="2c63df49-6605-4914-9756-a8dcb41d4a15" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.407556    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.440380    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.449396    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m" podUID="83f9f654-a050-4cdc-bc76-d2ea43698bdd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.449420    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.488700    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.501539    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.501649    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.626664    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.627566    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.627945    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.648398    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.672418    5845 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-dsdmp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.672479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.62:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.676378    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.693625    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.695406    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.882457    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:11.915392    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv" podUID="1eb30d9f-8aff-41d9-aa28-9fe8fc8e8912" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:12.714474    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:12.714569    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:12.726936    5845 trace.go:236] Trace[1226525525]: "iptables ChainExists" (01-Nov-2026 22:26:06.393) (total time: 6333ms):
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: Trace[1226525525]: [6.333507431s] [6.333507431s] END
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:12.726964    5845 trace.go:236] Trace[1115624419]: "iptables ChainExists" (01-Nov-2026 22:26:06.396) (total time: 6330ms):
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: Trace[1115624419]: [6.330890575s] [6.330890575s] END
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:13.725413    5845 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = network teardown failed for pod sandbox k8s_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr_openshift-ingress_6d07a690-ca3c-4d37-941d-68b7ef26f366_0(abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570): error removing pod openshift-ingress_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr from CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (delete): netplugin failed with no error message: signal: killed" podSandboxID="abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:13.725480    5845 kuberuntime_manager.go:1939] "Failed to stop sandbox" podSandboxID={"Type":"cri-o","ID":"abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570"}
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:13.835470    5845 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1m4.982s"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:13.835505    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:14.730229    5845 kubelet.go:2243] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"6d07a690-ca3c-4d37-941d-68b7ef26f366\" with KillPodSandboxError: \"rpc error: code = Unknown desc = network teardown failed for pod sandbox k8s_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr_openshift-ingress_6d07a690-ca3c-4d37-941d-68b7ef26f366_0(abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570): error removing pod openshift-ingress_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr from CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (delete): netplugin failed with no error message: signal: killed\"" logger="UnhandledError"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:14.730555    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"6d07a690-ca3c-4d37-941d-68b7ef26f366\" with KillPodSandboxError: \"rpc error: code = Unknown desc = network teardown failed for pod sandbox k8s_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr_openshift-ingress_6d07a690-ca3c-4d37-941d-68b7ef26f366_0(abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570): error removing pod openshift-ingress_payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr from CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (delete): netplugin failed with no error message: signal: killed\"" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:14.759389    5845 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-98dc2 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:14.759456    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:14.764280    5845 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" podUID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/healthz\": read tcp 10.134.0.2:55556->10.134.0.25:8081: read: connection reset by peer"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.083676    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.083731    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.085095    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.085154    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.089528    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.111868    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-09656e-dockercfg-6j5mv\""
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.111953    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-09656e\""
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128503    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128597    5845 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128620    5845 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128674    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128719    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128762    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128797    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-75d8948445-6nwm2"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128834    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kwlgq"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128852    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128878    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128919    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128955    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.128997    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.129042    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.129075    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.129989    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.132504    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92xm\" (UniqueName: \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.132710    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.148854    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:26:16.913376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.148905    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.148939    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.148972    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149004    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149035    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149050    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149100    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149125    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149142    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149166    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149199    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149227    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149257    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149285    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149330    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149361    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149402    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fszvp"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149437    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149463    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149492    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149520    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149535    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149562    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149604    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.149642    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.172668    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cdk5r"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.184915    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-x99jv"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.185478    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.189114    5845 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="metrics-server" containerStatusID={"Type":"cri-o","ID":"5e252301f4bb73c5f044f884e505e8b02bd5de82b66c1dbb906e59c314e11ae4"} pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" containerMessage="Container metrics-server failed liveness probe, will be restarted"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.189241    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" containerID="cri-o://5e252301f4bb73c5f044f884e505e8b02bd5de82b66c1dbb906e59c314e11ae4" gracePeriod=170
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.238722    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92xm\" (UniqueName: \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.238794    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.260370    5845 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"2cfd9a8996e696a5c2e1af6dad43dd2ca3312f1a4d84b973237769ceb9125624"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" containerMessage="Container csi-driver failed liveness probe, will be restarted"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.260542    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" podUID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerName="csi-driver" containerID="cri-o://2cfd9a8996e696a5c2e1af6dad43dd2ca3312f1a4d84b973237769ceb9125624" gracePeriod=30
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:16.931195    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:16.966238    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:17.012523    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.058207    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.168830    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\" (UniqueName: \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\") pod \"6d07a690-ca3c-4d37-941d-68b7ef26f366\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") "
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.168935    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\") pod \"6d07a690-ca3c-4d37-941d-68b7ef26f366\" (UID: \"6d07a690-ca3c-4d37-941d-68b7ef26f366\") "
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.169310    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume" pod "6d07a690-ca3c-4d37-941d-68b7ef26f366" (UID: "6d07a690-ca3c-4d37-941d-68b7ef26f366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.169770    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d07a690-ca3c-4d37-941d-68b7ef26f366-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.171443    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.171444    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.171519    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.171479    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.188251    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92xm\" (UniqueName: \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\") pod \"payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.216229    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.285708    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.285733    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr" event={"ID":"6d07a690-ca3c-4d37-941d-68b7ef26f366","Type":"ContainerDied","Data":"abcde0a3dff5205679f3896b660d1268299e3cdc5c6667ed4fca6ed225057570"}
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.285804    5845 scope.go:122] "RemoveContainer" containerID="e299590a7f7aa3056c59f43cebaa1fe18d25d709a8c45b706097fc488774c89c"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.299210    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4" pod "6d07a690-ca3c-4d37-941d-68b7ef26f366" (UID: "6d07a690-ca3c-4d37-941d-68b7ef26f366"). InnerVolumeSpecName "kube-api-access-jp8g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.308003    5845 trace.go:236] Trace[1098118726]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-98c48f84c-jd65m" (01-Nov-2026 22:26:06.693) (total time: 10614ms):
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: Trace[1098118726]: [10.614417695s] [10.614417695s] END
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.307997    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.308131    5845 generic.go:356] "Generic (PLEG): container finished" podID="98dd454d-3bb0-48d9-b6ed-d907da241372" containerID="2cfd9a8996e696a5c2e1af6dad43dd2ca3312f1a4d84b973237769ceb9125624" exitCode=2
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.308212    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerDied","Data":"2cfd9a8996e696a5c2e1af6dad43dd2ca3312f1a4d84b973237769ceb9125624"}
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.310478    5845 generic.go:356] "Generic (PLEG): container finished" podID="d1f29aa3-d6a1-4153-b759-0b130813ebf0" containerID="c6f054f7c58064d67d1e60250330fe205e80f404724fc452a69844379ebb315a" exitCode=1
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.310524    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" event={"ID":"d1f29aa3-d6a1-4153-b759-0b130813ebf0","Type":"ContainerDied","Data":"c6f054f7c58064d67d1e60250330fe205e80f404724fc452a69844379ebb315a"}
Nov 01 22:26:17.328583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.311095    5845 scope.go:122] "RemoveContainer" containerID="c6f054f7c58064d67d1e60250330fe205e80f404724fc452a69844379ebb315a"
Nov 01 22:26:17.490292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.381425    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp8g4\" (UniqueName: \"kubernetes.io/projected/6d07a690-ca3c-4d37-941d-68b7ef26f366-kube-api-access-jp8g4\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:17.490292 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.385204    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:26:17.629526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.619011    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:26:17.665311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.629535    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-829cb0-8f6cc8d47-f44nr"]
Nov 01 22:26:17.744820 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:17.744791    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd55f82_51b5_4137_8ed6_f7ef27e668fd.slice/crio-56af49c0967f0e74366a0cb9c49029b646287696f432ebbae0f46361ceb58fe3 WatchSource:0}: Error finding container 56af49c0967f0e74366a0cb9c49029b646287696f432ebbae0f46361ceb58fe3: Status 404 returned error can't find the container with id 56af49c0967f0e74366a0cb9c49029b646287696f432ebbae0f46361ceb58fe3
Nov 01 22:26:17.796608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.796573    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:17.796762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.796672    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:17.796762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.796714    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:17.972857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:17.972824    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 22:26:18.212521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.212415    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:18.212521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.212483    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:18.213442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.213405    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:26:18.213567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.213461    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:26:18.324772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.324737    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" event={"ID":"5dd55f82-51b5-4137-8ed6-f7ef27e668fd","Type":"ContainerStarted","Data":"56af49c0967f0e74366a0cb9c49029b646287696f432ebbae0f46361ceb58fe3"}
Nov 01 22:26:18.472799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.472770    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:26:18.532889 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:18.532858    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d36006_6cbe_4f59_aac1_5b1302e5beb2.slice/crio-04988f6b0518bdfc73e9aacef0f90d26b5b548440b33771441d0a04cce2de2a0 WatchSource:0}: Error finding container 04988f6b0518bdfc73e9aacef0f90d26b5b548440b33771441d0a04cce2de2a0: Status 404 returned error can't find the container with id 04988f6b0518bdfc73e9aacef0f90d26b5b548440b33771441d0a04cce2de2a0
Nov 01 22:26:18.871041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:18.870375    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" path="/var/lib/kubelet/pods/6d07a690-ca3c-4d37-941d-68b7ef26f366/volumes"
Nov 01 22:26:19.345847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.345750    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" event={"ID":"5dd55f82-51b5-4137-8ed6-f7ef27e668fd","Type":"ContainerStarted","Data":"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"}
Nov 01 22:26:19.349063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.349029    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct" event={"ID":"d1f29aa3-d6a1-4153-b759-0b130813ebf0","Type":"ContainerStarted","Data":"ab32c33486fe216588cf14bf5edaf2682dd3cade40230e06194632e65406435b"}
Nov 01 22:26:19.350194 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.350121    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 22:26:19.351750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.351716    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" event={"ID":"07d36006-6cbe-4f59-aac1-5b1302e5beb2","Type":"ContainerStarted","Data":"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"}
Nov 01 22:26:19.351867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.351781    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" event={"ID":"07d36006-6cbe-4f59-aac1-5b1302e5beb2","Type":"ContainerStarted","Data":"04988f6b0518bdfc73e9aacef0f90d26b5b548440b33771441d0a04cce2de2a0"}
Nov 01 22:26:19.351936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.351893    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:19.375066 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.375020    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podStartSLOduration=74.375006102 podStartE2EDuration="1m14.375006102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:19.370668019 +0000 UTC m=+2482.626847799" watchObservedRunningTime="2026-11-01 22:26:19.375006102 +0000 UTC m=+2482.631185864"
Nov 01 22:26:19.394134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:19.394081    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" podStartSLOduration=66.394063339 podStartE2EDuration="1m6.394063339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:19.392148892 +0000 UTC m=+2482.648328652" watchObservedRunningTime="2026-11-01 22:26:19.394063339 +0000 UTC m=+2482.650243103"
Nov 01 22:26:20.125703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:20.125670    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:20.127827 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:20.127792    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:20.127977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:20.127843    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:21.091386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.089963    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing" containerID="cri-o://1aa19a93aea78a20c14527807ea35d805bf9e150d870bfb3cf6cbc1e53d0609f" gracePeriod=25
Nov 01 22:26:21.124584 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.124525    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing" containerID="cri-o://c0f561df56a980cf2a6722ebce9c6c8a34c6ff42cee89297f47e6e6d1bf3649a" gracePeriod=25
Nov 01 22:26:21.125680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.125654    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:21.125761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.125715    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:21.580468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.424625    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:26:21.580468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:21.428467    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-98dc2" event={"ID":"98dd454d-3bb0-48d9-b6ed-d907da241372","Type":"ContainerStarted","Data":"459ade5d7f475c146a57fe6e2cc8bf04e2e5c505076dc7c0f3db486f7f7d8310"}
Nov 01 22:26:22.126822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.125431    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:22.126822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.126388    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:22.126822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.126451    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:22.523007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.522345    5845 generic.go:356] "Generic (PLEG): container finished" podID="ba174d90-0313-497b-a8e8-7d037efd859d" containerID="c0f561df56a980cf2a6722ebce9c6c8a34c6ff42cee89297f47e6e6d1bf3649a" exitCode=0
Nov 01 22:26:22.523007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.522579    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" event={"ID":"ba174d90-0313-497b-a8e8-7d037efd859d","Type":"ContainerDied","Data":"c0f561df56a980cf2a6722ebce9c6c8a34c6ff42cee89297f47e6e6d1bf3649a"}
Nov 01 22:26:22.524417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.524306    5845 generic.go:356] "Generic (PLEG): container finished" podID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerID="1aa19a93aea78a20c14527807ea35d805bf9e150d870bfb3cf6cbc1e53d0609f" exitCode=0
Nov 01 22:26:22.525133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.525111    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" event={"ID":"a8f7df48-8487-496a-a5c5-743eb09f9fa2","Type":"ContainerDied","Data":"1aa19a93aea78a20c14527807ea35d805bf9e150d870bfb3cf6cbc1e53d0609f"}
Nov 01 22:26:22.525394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.525155    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv" event={"ID":"a8f7df48-8487-496a-a5c5-743eb09f9fa2","Type":"ContainerDied","Data":"47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7"}
Nov 01 22:26:22.525394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.525173    5845 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e76d8c8e191198594b49439b35c313d3bed6486636f4374a659bd3009a55a7"
Nov 01 22:26:22.598546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.597004    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:26:22.605361 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.604753    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.632397    5845 patch_prober.go:29] interesting pod/metrics-server-6f99686fb5-2wbs5 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: [+]log ok
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: [+]poststarthook/max-in-flight-filter ok
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: [-]metric-collection-timely failed: reason withheld
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: [+]metadata-informer-sync ok
Nov 01 22:26:22.632427 ip-10-0-135-146 kubenswrapper[5845]: livez check failed
Nov 01 22:26:22.632747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.632447    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" podUID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 01 22:26:22.715367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.713988    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\") pod \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") "
Nov 01 22:26:22.715367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.714075    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\" (UniqueName: \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\") pod \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\" (UID: \"a8f7df48-8487-496a-a5c5-743eb09f9fa2\") "
Nov 01 22:26:22.715367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.714290    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\" (UniqueName: \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\") pod \"ba174d90-0313-497b-a8e8-7d037efd859d\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") "
Nov 01 22:26:22.715367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.714391    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\") pod \"ba174d90-0313-497b-a8e8-7d037efd859d\" (UID: \"ba174d90-0313-497b-a8e8-7d037efd859d\") "
Nov 01 22:26:22.719351 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.719320    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj" pod "ba174d90-0313-497b-a8e8-7d037efd859d" (UID: "ba174d90-0313-497b-a8e8-7d037efd859d"). InnerVolumeSpecName "kube-api-access-crqtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:22.790450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.738153    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume" pod "ba174d90-0313-497b-a8e8-7d037efd859d" (UID: "ba174d90-0313-497b-a8e8-7d037efd859d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:22.819071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.818982    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crqtj\" (UniqueName: \"kubernetes.io/projected/ba174d90-0313-497b-a8e8-7d037efd859d-kube-api-access-crqtj\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:22.819071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.819065    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba174d90-0313-497b-a8e8-7d037efd859d-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:22.823843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.823797    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume" pod "a8f7df48-8487-496a-a5c5-743eb09f9fa2" (UID: "a8f7df48-8487-496a-a5c5-743eb09f9fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:22.825659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.825623    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb" pod "a8f7df48-8487-496a-a5c5-743eb09f9fa2" (UID: "a8f7df48-8487-496a-a5c5-743eb09f9fa2"). InnerVolumeSpecName "kube-api-access-sx8cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:22.928126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.920663    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f7df48-8487-496a-a5c5-743eb09f9fa2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:22.928126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:22.920688    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8cb\" (UniqueName: \"kubernetes.io/projected/a8f7df48-8487-496a-a5c5-743eb09f9fa2-kube-api-access-sx8cb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:23.125667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.125635    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:23.125895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.125690    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:23.533260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.533230    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"
Nov 01 22:26:23.533813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.533795    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"
Nov 01 22:26:23.533929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.533801    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2" event={"ID":"ba174d90-0313-497b-a8e8-7d037efd859d","Type":"ContainerDied","Data":"2c5a8c2c53a902a94653258350ac7ca06a5220a828f49e9d2adcaaee11c173aa"}
Nov 01 22:26:23.533929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.533892    5845 scope.go:122] "RemoveContainer" containerID="c0f561df56a980cf2a6722ebce9c6c8a34c6ff42cee89297f47e6e6d1bf3649a"
Nov 01 22:26:23.588529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.588008    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:26:23.595720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.595680    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0bb375-5f666dc5ff-gh7q2"]
Nov 01 22:26:23.609040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.609011    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:26:23.622404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:23.622381    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0bb375-5bb6579594-5qbkv"]
Nov 01 22:26:24.126824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:24.125957    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:24.126824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:24.126057    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:24.882968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:24.882934    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" path="/var/lib/kubelet/pods/a8f7df48-8487-496a-a5c5-743eb09f9fa2/volumes"
Nov 01 22:26:24.915916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:24.883448    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" path="/var/lib/kubelet/pods/ba174d90-0313-497b-a8e8-7d037efd859d/volumes"
Nov 01 22:26:25.126135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:25.126101    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:26:25.126341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:25.126153    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 01 22:26:25.749274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:25.749237    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:26:26.130818 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:26.130730    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:26.131955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:26.131932    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:28.342998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.342961    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:26:28.343963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.343940    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing"
Nov 01 22:26:28.343963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.343962    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing"
Nov 01 22:26:28.344130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.343995    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing"
Nov 01 22:26:28.344130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344002    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing"
Nov 01 22:26:28.344130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344013    5845 cpu_manager.go:414] "containerMap: removing container" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing"
Nov 01 22:26:28.344130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344018    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing"
Nov 01 22:26:28.344260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344150    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d07a690-ca3c-4d37-941d-68b7ef26f366" containerName="payload-pre-processing"
Nov 01 22:26:28.344260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344167    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba174d90-0313-497b-a8e8-7d037efd859d" containerName="payload-pre-processing"
Nov 01 22:26:28.344260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.344178    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f7df48-8487-496a-a5c5-743eb09f9fa2" containerName="payload-processing"
Nov 01 22:26:28.444280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.444248    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.448249 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.448205    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2a68ac-openshift-default-dockercfg-lw2cp\""
Nov 01 22:26:28.481025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.480995    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481181 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481090    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481189    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481276    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481379    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481519 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481489    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.481713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481690    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.487428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481842    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27kf\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.487428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.481883    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.560635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.560285    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:26:28.591232 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.591196    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.591512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.591491    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.591652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.591635    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.598128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.591779    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.598342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.598318    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.598741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.598714    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.599040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.599025    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27kf\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.599168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.599155    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.604358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.600029    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.604358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.600314    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.604358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.600567    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.604358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.600655    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.604736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.604711    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.608942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.607500    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.704164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.703134    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.704394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.704222    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.748834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.748796    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27kf\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.759545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.749815    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\") pod \"e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") " pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:28.778022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:28.777647    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:29.148138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.148111    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:26:29.149998 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:29.149972    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006371a8_42ac_402e_ba6c_8bd9e13ed5d8.slice/crio-12e6936efb0ae3e1ec736efe92c3a8da503698a44116830b4075a77082df695d WatchSource:0}: Error finding container 12e6936efb0ae3e1ec736efe92c3a8da503698a44116830b4075a77082df695d: Status 404 returned error can't find the container with id 12e6936efb0ae3e1ec736efe92c3a8da503698a44116830b4075a77082df695d
Nov 01 22:26:29.153522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.153481    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:29.153600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.153578    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:29.153645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.153620    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:29.628245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.628206    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerStarted","Data":"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"}
Nov 01 22:26:29.628776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.628276    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerStarted","Data":"12e6936efb0ae3e1ec736efe92c3a8da503698a44116830b4075a77082df695d"}
Nov 01 22:26:29.658597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.658541    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podStartSLOduration=1.6585222160000002 podStartE2EDuration="1.658522216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:29.653839151 +0000 UTC m=+2492.910018926" watchObservedRunningTime="2026-11-01 22:26:29.658522216 +0000 UTC m=+2492.914701980"
Nov 01 22:26:29.793989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:29.791394    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:30.383596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.383513    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:26:30.640501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.640410    5845 generic.go:356] "Generic (PLEG): container finished" podID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerID="78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459" exitCode=0
Nov 01 22:26:30.640975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.640601    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerDied","Data":"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"}
Nov 01 22:26:30.641280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.641253    5845 scope.go:122] "RemoveContainer" containerID="78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"
Nov 01 22:26:30.645318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.645061    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.645318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.645212    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.645509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.645327    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.649340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.649262    5845 generic.go:356] "Generic (PLEG): container finished" podID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerID="b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1" exitCode=0
Nov 01 22:26:30.649437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.649334    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerDied","Data":"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"}
Nov 01 22:26:30.650048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.650030    5845 scope.go:122] "RemoveContainer" containerID="b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"
Nov 01 22:26:30.652649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.652040    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.652649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.652105    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.652649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.652141    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:26:30.799544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:30.799521    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:31.660137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:31.660103    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerStarted","Data":"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"}
Nov 01 22:26:31.662357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:31.662323    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerStarted","Data":"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"}
Nov 01 22:26:31.662522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:31.662507    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:31.663630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:31.663599    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:26:32.102038 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.101254    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:26:32.102537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.102514    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:32.102732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.102709    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:32.186325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.183215    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:26:32.186325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.183317    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:26:32.331643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.331604    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:26:32.332286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.332257    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:32.332456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:32.332348    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:33.104380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.101139    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:33.104380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.101200    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:33.211888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.211819    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:26:33.223192 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.223168    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:26:33.223539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.223475    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.312174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.312133    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.312383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.312183    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65lv\" (UniqueName: \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.320582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.320558    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a68ac\""
Nov 01 22:26:33.331473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.331448    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:33.331609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.331501    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:33.430373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.423242    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.430373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.423331    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65lv\" (UniqueName: \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.469353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.442039    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.469353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.451137    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65lv\" (UniqueName: \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\") pod \"payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:33.546320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:33.545559    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:34.260388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.158449    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:34.260388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.158529    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:34.366135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.365988    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:34.368435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.368380    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:34.382634 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.382590    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:26:34.442355 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:34.434165    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edc5e52_e102_496d_a562_8278e9bc2354.slice/crio-72f7c444b772215fe9885cfe305d03bb90bdcad5fa8745622d6f44d4becc3335 WatchSource:0}: Error finding container 72f7c444b772215fe9885cfe305d03bb90bdcad5fa8745622d6f44d4becc3335: Status 404 returned error can't find the container with id 72f7c444b772215fe9885cfe305d03bb90bdcad5fa8745622d6f44d4becc3335
Nov 01 22:26:34.837993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.837891    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" event={"ID":"8edc5e52-e102-496d-a562-8278e9bc2354","Type":"ContainerStarted","Data":"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"}
Nov 01 22:26:34.838194 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.838034    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" event={"ID":"8edc5e52-e102-496d-a562-8278e9bc2354","Type":"ContainerStarted","Data":"72f7c444b772215fe9885cfe305d03bb90bdcad5fa8745622d6f44d4becc3335"}
Nov 01 22:26:34.838273 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.838253    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:34.905230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:34.904879    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" podStartSLOduration=1.9048598700000001 podStartE2EDuration="1.90485987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:34.866771839 +0000 UTC m=+2498.122951615" watchObservedRunningTime="2026-11-01 22:26:34.90485987 +0000 UTC m=+2498.161039635"
Nov 01 22:26:34.917794 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:34.917752    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:26:35.102375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:35.101816    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:35.102375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:35.101877    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:35.332086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:35.332055    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:35.332582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:35.332352    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:36.101157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:36.101111    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:36.101377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:36.101170    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:36.331434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:36.331400    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:36.331608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:36.331457    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:37.101092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:37.101057    5845 patch_prober.go:29] interesting pod/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 01 22:26:37.101704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:37.101666    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 01 22:26:37.331940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:37.331901    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body=
Nov 01 22:26:37.332133 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:37.331963    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused"
Nov 01 22:26:38.149968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:38.149922    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:26:38.151837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:38.151797    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:26:38.342389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:38.342351    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:26:38.343566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:38.343542    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:26:39.106156 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.106122    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:26:39.111110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.111087    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.116567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.116544    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-56452a\""
Nov 01 22:26:39.116882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.116860    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-56452a-dockercfg-fscqg\""
Nov 01 22:26:39.121591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.121285    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:26:39.185496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.185459    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6m5m\" (UniqueName: \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.185981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.185772    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.295573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.295528    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6m5m\" (UniqueName: \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.296147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.296121    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.297262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.297238    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.373085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.373007    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6m5m\" (UniqueName: \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\") pod \"payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.433691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.431616    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.752795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.752768    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:26:39.754514 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:26:39.754484    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b21d62_8ea5_4d4c_add3_7ffa25eb7388.slice/crio-c5a45cfcae58bdf6eb3150e0fd83ffa4428dd9f94a93b6f81d6399aac575be11 WatchSource:0}: Error finding container c5a45cfcae58bdf6eb3150e0fd83ffa4428dd9f94a93b6f81d6399aac575be11: Status 404 returned error can't find the container with id c5a45cfcae58bdf6eb3150e0fd83ffa4428dd9f94a93b6f81d6399aac575be11
Nov 01 22:26:39.936707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.936669    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" event={"ID":"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388","Type":"ContainerStarted","Data":"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"}
Nov 01 22:26:39.936832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.936729    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" event={"ID":"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388","Type":"ContainerStarted","Data":"c5a45cfcae58bdf6eb3150e0fd83ffa4428dd9f94a93b6f81d6399aac575be11"}
Nov 01 22:26:39.936832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.936757    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:39.959112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:39.959051    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" podStartSLOduration=0.95903387 podStartE2EDuration="959.03387ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:26:39.954881401 +0000 UTC m=+2503.211061165" watchObservedRunningTime="2026-11-01 22:26:39.95903387 +0000 UTC m=+2503.215213637"
Nov 01 22:26:40.704522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:40.704483    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:26:40.705041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:40.704729    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerName="payload-processing" containerID="cri-o://a9b0512973b3bf47374301533afbaa05b0fcb70615116ffe534c81aac5696b79" gracePeriod=30
Nov 01 22:26:41.237459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.221874    5845 generic.go:356] "Generic (PLEG): container finished" podID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerID="a9b0512973b3bf47374301533afbaa05b0fcb70615116ffe534c81aac5696b79" exitCode=0
Nov 01 22:26:41.237459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.223619    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" event={"ID":"4eb26c1d-9648-4313-b54c-14c616894e4e","Type":"ContainerDied","Data":"a9b0512973b3bf47374301533afbaa05b0fcb70615116ffe534c81aac5696b79"}
Nov 01 22:26:41.273024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.272996    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:26:41.378272 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.378238    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\") pod \"4eb26c1d-9648-4313-b54c-14c616894e4e\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") "
Nov 01 22:26:41.378502 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.378351    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\" (UniqueName: \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\") pod \"4eb26c1d-9648-4313-b54c-14c616894e4e\" (UID: \"4eb26c1d-9648-4313-b54c-14c616894e4e\") "
Nov 01 22:26:41.378752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.378704    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume" pod "4eb26c1d-9648-4313-b54c-14c616894e4e" (UID: "4eb26c1d-9648-4313-b54c-14c616894e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:41.379289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.379267    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb26c1d-9648-4313-b54c-14c616894e4e-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:41.381509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.381478    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr" pod "4eb26c1d-9648-4313-b54c-14c616894e4e" (UID: "4eb26c1d-9648-4313-b54c-14c616894e4e"). InnerVolumeSpecName "kube-api-access-h68zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:41.433402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.433358    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dz5ct"
Nov 01 22:26:41.481973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:41.481935    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68zr\" (UniqueName: \"kubernetes.io/projected/4eb26c1d-9648-4313-b54c-14c616894e4e-kube-api-access-h68zr\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:42.231013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.230981    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"
Nov 01 22:26:42.231013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.230993    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb" event={"ID":"4eb26c1d-9648-4313-b54c-14c616894e4e","Type":"ContainerDied","Data":"68a7ab637453d98db28240f5156a1b512617f8fea9d31d397bb46c40a0088208"}
Nov 01 22:26:42.231570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.231068    5845 scope.go:122] "RemoveContainer" containerID="a9b0512973b3bf47374301533afbaa05b0fcb70615116ffe534c81aac5696b79"
Nov 01 22:26:42.258944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.258916    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:26:42.262714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.262686    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e25317-7bcfcd9d96-hfxtb"]
Nov 01 22:26:42.490962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.490874    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main" containerID="cri-o://2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f" gracePeriod=44
Nov 01 22:26:42.798470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.798441    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:26:42.865174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.865147    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" path="/var/lib/kubelet/pods/4eb26c1d-9648-4313-b54c-14c616894e4e/volumes"
Nov 01 22:26:42.905169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905137    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.905362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905228    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.905362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905288    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.905456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905387    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.905456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905449    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.906102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905481    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\" (UniqueName: \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\") pod \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\" (UID: \"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8\") "
Nov 01 22:26:42.906102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905585    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:42.906102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905613    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:42.906102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.905727    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:42.906102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.906088    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:42.906356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.906120    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:42.906356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.906129    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:42.907656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.907631    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:42.907779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.907681    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:26:42.908030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:42.908010    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr" pod "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" (UID: "90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8"). InnerVolumeSpecName "kube-api-access-xjdwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:43.007269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.007245    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:43.007269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.007268    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:43.007423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.007278    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdwr\" (UniqueName: \"kubernetes.io/projected/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8-kube-api-access-xjdwr\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:43.239916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.239886    5845 generic.go:356] "Generic (PLEG): container finished" podID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerID="2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f" exitCode=0
Nov 01 22:26:43.240394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.239925    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"
Nov 01 22:26:43.240394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.239970    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" event={"ID":"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8","Type":"ContainerDied","Data":"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"}
Nov 01 22:26:43.240394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.240011    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw" event={"ID":"90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8","Type":"ContainerDied","Data":"f748c3adb2337bf96d6bc24d3d2789148f23ccab37e70f9cd4439ec491bebd4c"}
Nov 01 22:26:43.240394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.240032    5845 scope.go:122] "RemoveContainer" containerID="2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"
Nov 01 22:26:43.253406 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.253389    5845 scope.go:122] "RemoveContainer" containerID="2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"
Nov 01 22:26:43.253922 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:43.253904    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f\": container with ID starting with 2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f not found: ID does not exist" containerID="2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"
Nov 01 22:26:43.254950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.253933    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f"} err="failed to get container status \"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f\": rpc error: code = NotFound desc = could not find container \"2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f\": container with ID starting with 2c49eb399d1e6a95110c70adb54433a583f177f2b81d0206529fa688ec195f1f not found: ID does not exist"
Nov 01 22:26:43.264705 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.264683    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:26:43.272789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:43.272767    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-f758d9/rate-test-model-f758d9-kserve-7b7bdbb8fc-gt9nw"]
Nov 01 22:26:44.864162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:44.864122    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" path="/var/lib/kubelet/pods/90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8/volumes"
Nov 01 22:26:45.921879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:45.919855    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:26:48.954634 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:48.954606    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:26:48.955143 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:48.954832    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy" containerID="cri-o://30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e" gracePeriod=30
Nov 01 22:26:48.961738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:48.961705    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:26:48.963382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:48.963346    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy" containerID="cri-o://324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2" gracePeriod=30
Nov 01 22:26:51.226243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:51.226207    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:26:54.294924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.294893    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:26:54.390740 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.390710    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:54.395403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.395375    5845 generic.go:356] "Generic (PLEG): container finished" podID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerID="324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2" exitCode=0
Nov 01 22:26:54.395537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.395507    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" event={"ID":"5dd55f82-51b5-4137-8ed6-f7ef27e668fd","Type":"ContainerDied","Data":"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"}
Nov 01 22:26:54.395617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.395587    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"
Nov 01 22:26:54.395678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.395616    5845 scope.go:122] "RemoveContainer" containerID="324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"
Nov 01 22:26:54.395737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.395593    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t" event={"ID":"5dd55f82-51b5-4137-8ed6-f7ef27e668fd","Type":"ContainerDied","Data":"56af49c0967f0e74366a0cb9c49029b646287696f432ebbae0f46361ceb58fe3"}
Nov 01 22:26:54.398409 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.398360    5845 generic.go:356] "Generic (PLEG): container finished" podID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerID="30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e" exitCode=0
Nov 01 22:26:54.398516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.398457    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"
Nov 01 22:26:54.398633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.398610    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerDied","Data":"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"}
Nov 01 22:26:54.398692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.398658    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz" event={"ID":"a1e13660-8961-418b-afd7-1bdf2ee53d34","Type":"ContainerDied","Data":"985f7ddbb21abc0914082c03e51cd5d68cd345ea5947d4f82f84fde096e66f50"}
Nov 01 22:26:54.407490 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.407465    5845 scope.go:122] "RemoveContainer" containerID="324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"
Nov 01 22:26:54.407775 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:54.407753    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2\": container with ID starting with 324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2 not found: ID does not exist" containerID="324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"
Nov 01 22:26:54.407872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.407782    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2"} err="failed to get container status \"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2\": rpc error: code = NotFound desc = could not find container \"324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2\": container with ID starting with 324876d9dd32d04728c6abb2d1d235c10c39115b597c8965a6115d5219f503a2 not found: ID does not exist"
Nov 01 22:26:54.407872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.407798    5845 scope.go:122] "RemoveContainer" containerID="30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"
Nov 01 22:26:54.421019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.420987    5845 scope.go:122] "RemoveContainer" containerID="bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"
Nov 01 22:26:54.433237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.433219    5845 scope.go:122] "RemoveContainer" containerID="30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"
Nov 01 22:26:54.433542 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:54.433519    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e\": container with ID starting with 30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e not found: ID does not exist" containerID="30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"
Nov 01 22:26:54.435283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.433550    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e"} err="failed to get container status \"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e\": rpc error: code = NotFound desc = could not find container \"30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e\": container with ID starting with 30f4d1ecb8bc10ca88b22f7367c01687f606cb65826c3219a747424787f4f23e not found: ID does not exist"
Nov 01 22:26:54.435283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.433567    5845 scope.go:122] "RemoveContainer" containerID="bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"
Nov 01 22:26:54.435283 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:26:54.433879    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092\": container with ID starting with bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092 not found: ID does not exist" containerID="bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"
Nov 01 22:26:54.435283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.433912    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092"} err="failed to get container status \"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092\": rpc error: code = NotFound desc = could not find container \"bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092\": container with ID starting with bc6cf7e9936cc8546895824cbcf6ce7f7e74cf68e792e855cfb9bdf6d669e092 not found: ID does not exist"
Nov 01 22:26:54.436526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.436502    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.436826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.436807    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.436949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.436933    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.436960    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.436980    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437265    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437394    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437327    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437366    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437504    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437571    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437594    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.437661 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.437618    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\") pod \"a1e13660-8961-418b-afd7-1bdf2ee53d34\" (UID: \"a1e13660-8961-418b-afd7-1bdf2ee53d34\") "
Nov 01 22:26:54.438383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438359    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:54.438569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438546    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e13660-8961-418b-afd7-1bdf2ee53d34-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.438651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438588    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.438651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438602    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.438651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438615    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.438651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.438627    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.440141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.440105    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.441760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.441188    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:54.441969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.441947    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "kube-api-access-4wnt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:54.445832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.445805    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo" pod "a1e13660-8961-418b-afd7-1bdf2ee53d34" (UID: "a1e13660-8961-418b-afd7-1bdf2ee53d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:26:54.539620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539528    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.539620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539589    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.539805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539661    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.539805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539714    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.539897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539874    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.539993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539965    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:54.540039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.539973    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.540093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540045    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.540153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540105    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.540260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540204    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.540375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540339    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.541291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540370    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\") pod \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\" (UID: \"5dd55f82-51b5-4137-8ed6-f7ef27e668fd\") "
Nov 01 22:26:54.541291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540546    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.541291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.540640    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.541291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541277    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541323    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541340    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541353    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541367    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541381    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnt2\" (UniqueName: \"kubernetes.io/projected/a1e13660-8961-418b-afd7-1bdf2ee53d34-kube-api-access-4wnt2\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541396    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541411    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541424    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.541925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.541878    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:26:54.542170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.542143    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "kube-api-access-f9ggp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:54.542381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.542356    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:54.543394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.542399    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token" pod "5dd55f82-51b5-4137-8ed6-f7ef27e668fd" (UID: "5dd55f82-51b5-4137-8ed6-f7ef27e668fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:54.642532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.642502    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.642532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.642528    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.642532 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.642538    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.642829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.642553    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ggp\" (UniqueName: \"kubernetes.io/projected/5dd55f82-51b5-4137-8ed6-f7ef27e668fd-kube-api-access-f9ggp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:54.736220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.736188    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:26:54.740266 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.740236    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-tj44t"]
Nov 01 22:26:54.751836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.751805    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:26:54.756512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.756476    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e25317-openshift-default-58f4485c6f-72cxz"]
Nov 01 22:26:54.864377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.864286    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" path="/var/lib/kubelet/pods/5dd55f82-51b5-4137-8ed6-f7ef27e668fd/volumes"
Nov 01 22:26:54.864836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:26:54.864822    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" path="/var/lib/kubelet/pods/a1e13660-8961-418b-afd7-1bdf2ee53d34/volumes"
Nov 01 22:27:04.824241 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:04.824203    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:27:07.413086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:07.413054    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:27:07.413737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:07.413686    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" podUID="372ed1e0-298d-49c3-8b50-61c0116b3334" containerName="payload-pre-processing" containerID="cri-o://d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543" gracePeriod=30
Nov 01 22:27:08.035130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.035102    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:27:08.095387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.095353    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\") pod \"372ed1e0-298d-49c3-8b50-61c0116b3334\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") "
Nov 01 22:27:08.095814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.095791    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\" (UniqueName: \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\") pod \"372ed1e0-298d-49c3-8b50-61c0116b3334\" (UID: \"372ed1e0-298d-49c3-8b50-61c0116b3334\") "
Nov 01 22:27:08.100136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.100098    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl" pod "372ed1e0-298d-49c3-8b50-61c0116b3334" (UID: "372ed1e0-298d-49c3-8b50-61c0116b3334"). InnerVolumeSpecName "kube-api-access-vpzzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:08.100686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.100650    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume" pod "372ed1e0-298d-49c3-8b50-61c0116b3334" (UID: "372ed1e0-298d-49c3-8b50-61c0116b3334"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:08.199467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.199431    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372ed1e0-298d-49c3-8b50-61c0116b3334-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:08.199467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.199463    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpzzl\" (UniqueName: \"kubernetes.io/projected/372ed1e0-298d-49c3-8b50-61c0116b3334-kube-api-access-vpzzl\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:08.535914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.535880    5845 generic.go:356] "Generic (PLEG): container finished" podID="372ed1e0-298d-49c3-8b50-61c0116b3334" containerID="d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543" exitCode=0
Nov 01 22:27:08.536416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.535931    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"
Nov 01 22:27:08.536416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.535963    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" event={"ID":"372ed1e0-298d-49c3-8b50-61c0116b3334","Type":"ContainerDied","Data":"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"}
Nov 01 22:27:08.536416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.536062    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd" event={"ID":"372ed1e0-298d-49c3-8b50-61c0116b3334","Type":"ContainerDied","Data":"e88a335e621a17b744f7afc0b5a57e8127867bd0872a15c804a21df17a70d58c"}
Nov 01 22:27:08.536416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.536096    5845 scope.go:122] "RemoveContainer" containerID="d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"
Nov 01 22:27:08.553808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.553783    5845 scope.go:122] "RemoveContainer" containerID="d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"
Nov 01 22:27:08.554268 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:08.554236    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543\": container with ID starting with d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543 not found: ID does not exist" containerID="d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"
Nov 01 22:27:08.554397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.554273    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543"} err="failed to get container status \"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543\": rpc error: code = NotFound desc = could not find container \"d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543\": container with ID starting with d2e8195a7b70749d6e598a74cb03e8e2d02f78e731c75ae738de0038b0ccc543 not found: ID does not exist"
Nov 01 22:27:08.561479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.561459    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:27:08.565394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.565373    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad047b-646fb5b6c9-tj8wd"]
Nov 01 22:27:08.867199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:08.867117    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372ed1e0-298d-49c3-8b50-61c0116b3334" path="/var/lib/kubelet/pods/372ed1e0-298d-49c3-8b50-61c0116b3334/volumes"
Nov 01 22:27:15.207685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:15.207652    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:27:15.208206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:15.207944    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy" containerID="cri-o://568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb" gracePeriod=30
Nov 01 22:27:20.669488 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.669459    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:27:20.688223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688160    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688211    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688351    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688403    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688488    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688530    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688597    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688642    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.688803 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.688780    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\") pod \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\" (UID: \"a4527a39-2e72-421b-a3bd-c74ae73b08ea\") "
Nov 01 22:27:20.690384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690343    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:20.690511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690361    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:20.690630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690589    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:20.690725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690625    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "kube-api-access-lhfvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:20.690789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690724    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:20.690789 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.690767    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:20.692591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.692556    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:20.693163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.693146    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:20.694323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.694266    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo" pod "a4527a39-2e72-421b-a3bd-c74ae73b08ea" (UID: "a4527a39-2e72-421b-a3bd-c74ae73b08ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:27:20.791461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791425    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791460    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791473    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791485    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791497    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791508    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4527a39-2e72-421b-a3bd-c74ae73b08ea-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791519    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791531    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4527a39-2e72-421b-a3bd-c74ae73b08ea-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:20.791706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:20.791543    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfvw\" (UniqueName: \"kubernetes.io/projected/a4527a39-2e72-421b-a3bd-c74ae73b08ea-kube-api-access-lhfvw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:21.006492 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.006458    5845 generic.go:356] "Generic (PLEG): container finished" podID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerID="568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb" exitCode=0
Nov 01 22:27:21.006680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.006502    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"
Nov 01 22:27:21.006680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.006546    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerDied","Data":"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"}
Nov 01 22:27:21.006680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.006621    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws" event={"ID":"a4527a39-2e72-421b-a3bd-c74ae73b08ea","Type":"ContainerDied","Data":"47a0bdf6d793463c64bfa3cc43aa825bbef767ab55162966d0174d7a406f325b"}
Nov 01 22:27:21.006680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.006657    5845 scope.go:122] "RemoveContainer" containerID="568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"
Nov 01 22:27:21.019152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.019131    5845 scope.go:122] "RemoveContainer" containerID="78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"
Nov 01 22:27:21.037481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.037453    5845 scope.go:122] "RemoveContainer" containerID="568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"
Nov 01 22:27:21.038898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.038873    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:27:21.039799 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:21.039019    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb\": container with ID starting with 568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb not found: ID does not exist" containerID="568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"
Nov 01 22:27:21.039799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.039048    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb"} err="failed to get container status \"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb\": rpc error: code = NotFound desc = could not find container \"568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb\": container with ID starting with 568533e8b8fbb1ca8012a5aba7620d00c19d6073ccb4fb9864a93eda163582bb not found: ID does not exist"
Nov 01 22:27:21.039799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.039070    5845 scope.go:122] "RemoveContainer" containerID="78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"
Nov 01 22:27:21.039799 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:21.039420    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459\": container with ID starting with 78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459 not found: ID does not exist" containerID="78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"
Nov 01 22:27:21.039799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.039443    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459"} err="failed to get container status \"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459\": rpc error: code = NotFound desc = could not find container \"78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459\": container with ID starting with 78536c851acfe785b1685d9332401b0e18f2f2523121002f89d0bbb178c6c459 not found: ID does not exist"
Nov 01 22:27:21.052498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.052465    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad047b-openshift-default-76dc48b85-cssws"]
Nov 01 22:27:21.305128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.305087    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:27:21.306314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306271    5845 cpu_manager.go:414] "containerMap: removing container" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerName="payload-processing"
Nov 01 22:27:21.306314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306315    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerName="payload-processing"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306334    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306343    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306359    5845 cpu_manager.go:414] "containerMap: removing container" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306367    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306397    5845 cpu_manager.go:414] "containerMap: removing container" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306404    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306434    5845 cpu_manager.go:414] "containerMap: removing container" podUID="372ed1e0-298d-49c3-8b50-61c0116b3334" containerName="payload-pre-processing"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306441    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372ed1e0-298d-49c3-8b50-61c0116b3334" containerName="payload-pre-processing"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306466    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.306501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306473    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306508    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306515    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306683    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306700    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb26c1d-9648-4313-b54c-14c616894e4e" containerName="payload-processing"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306712    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306721    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a0896c-8c14-47ac-96fa-4b0fe5f8d7d8" containerName="main"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306735    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd55f82-51b5-4137-8ed6-f7ef27e668fd" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306748    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="372ed1e0-298d-49c3-8b50-61c0116b3334" containerName="payload-pre-processing"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306761    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:21.307041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.306770    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" containerName="istio-proxy"
Nov 01 22:27:21.319013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.318986    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.329545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.329513    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a00c6a\""
Nov 01 22:27:21.341700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.341671    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:27:21.404398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.404364    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.404579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.404560    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4r45\" (UniqueName: \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.505592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.505551    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.505771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.505717    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4r45\" (UniqueName: \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.506417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.506392    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.518263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.518235    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4r45\" (UniqueName: \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\") pod \"payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:21.657022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:21.656947    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:22.134049 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:22.134004    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:27:22.148188 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:22.146585    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99683f0_28b4_48cd_9b26_6a386ba7d1f2.slice/crio-aaf018287890d8b066f340b0ccf3aa14234fc56cb9ed8dd55fc12e751c7d12bf WatchSource:0}: Error finding container aaf018287890d8b066f340b0ccf3aa14234fc56cb9ed8dd55fc12e751c7d12bf: Status 404 returned error can't find the container with id aaf018287890d8b066f340b0ccf3aa14234fc56cb9ed8dd55fc12e751c7d12bf
Nov 01 22:27:22.697320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:22.692373    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:27:22.867231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:22.867190    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4527a39-2e72-421b-a3bd-c74ae73b08ea" path="/var/lib/kubelet/pods/a4527a39-2e72-421b-a3bd-c74ae73b08ea/volumes"
Nov 01 22:27:23.202323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:23.201424    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" event={"ID":"e99683f0-28b4-48cd-9b26-6a386ba7d1f2","Type":"ContainerStarted","Data":"caf51b41cf910d5f705653e393d8746e412399a668b7ca434a1c440370619348"}
Nov 01 22:27:23.202323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:23.201506    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" event={"ID":"e99683f0-28b4-48cd-9b26-6a386ba7d1f2","Type":"ContainerStarted","Data":"aaf018287890d8b066f340b0ccf3aa14234fc56cb9ed8dd55fc12e751c7d12bf"}
Nov 01 22:27:23.202893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:23.202655    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:23.248354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:23.248235    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" podStartSLOduration=2.248219202 podStartE2EDuration="2.248219202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:23.245713658 +0000 UTC m=+2546.501893422" watchObservedRunningTime="2026-11-01 22:27:23.248219202 +0000 UTC m=+2546.504398943"
Nov 01 22:27:24.897832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:24.897796    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:27:27.692813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:27.692772    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" podUID="8edc5e52-e102-496d-a562-8278e9bc2354" containerName="payload-processing" containerID="cri-o://f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20" gracePeriod=25
Nov 01 22:27:28.009027 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.009000    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:27:28.103749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.103715    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\" (UniqueName: \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\") pod \"8edc5e52-e102-496d-a562-8278e9bc2354\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") "
Nov 01 22:27:28.103941 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.103846    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\") pod \"8edc5e52-e102-496d-a562-8278e9bc2354\" (UID: \"8edc5e52-e102-496d-a562-8278e9bc2354\") "
Nov 01 22:27:28.104816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.104243    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume" pod "8edc5e52-e102-496d-a562-8278e9bc2354" (UID: "8edc5e52-e102-496d-a562-8278e9bc2354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:28.104816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.104498    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc5e52-e102-496d-a562-8278e9bc2354-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:28.105798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.105772    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv" pod "8edc5e52-e102-496d-a562-8278e9bc2354" (UID: "8edc5e52-e102-496d-a562-8278e9bc2354"). InnerVolumeSpecName "kube-api-access-v65lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:28.206381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.206312    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65lv\" (UniqueName: \"kubernetes.io/projected/8edc5e52-e102-496d-a562-8278e9bc2354-kube-api-access-v65lv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:28.316548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.316475    5845 generic.go:356] "Generic (PLEG): container finished" podID="8edc5e52-e102-496d-a562-8278e9bc2354" containerID="f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20" exitCode=0
Nov 01 22:27:28.316548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.316525    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"
Nov 01 22:27:28.316548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.316562    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" event={"ID":"8edc5e52-e102-496d-a562-8278e9bc2354","Type":"ContainerDied","Data":"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"}
Nov 01 22:27:28.316858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.316629    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56" event={"ID":"8edc5e52-e102-496d-a562-8278e9bc2354","Type":"ContainerDied","Data":"72f7c444b772215fe9885cfe305d03bb90bdcad5fa8745622d6f44d4becc3335"}
Nov 01 22:27:28.316858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.316652    5845 scope.go:122] "RemoveContainer" containerID="f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"
Nov 01 22:27:28.328232 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.328211    5845 scope.go:122] "RemoveContainer" containerID="f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"
Nov 01 22:27:28.328559 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:28.328538    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20\": container with ID starting with f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20 not found: ID does not exist" containerID="f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"
Nov 01 22:27:28.328633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.328569    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20"} err="failed to get container status \"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20\": rpc error: code = NotFound desc = could not find container \"f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20\": container with ID starting with f1c69642f842673beb648135182ea281d21d0c59009f23a00a151992b9208d20 not found: ID does not exist"
Nov 01 22:27:28.341547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.341514    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:27:28.346116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.346096    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-6cb788b5db-87m56"]
Nov 01 22:27:28.547659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.547621    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:27:28.863240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:28.863163    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edc5e52-e102-496d-a562-8278e9bc2354" path="/var/lib/kubelet/pods/8edc5e52-e102-496d-a562-8278e9bc2354/volumes"
Nov 01 22:27:29.907318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:29.906608    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing" containerID="cri-o://0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224" gracePeriod=25
Nov 01 22:27:33.548515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.548463    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" podUID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" containerName="payload-processing" containerID="cri-o://0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b" gracePeriod=25
Nov 01 22:27:33.906416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.906387    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:27:33.967526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.967496    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\" (UniqueName: \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\") pod \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") "
Nov 01 22:27:33.967715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.967693    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\") pod \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\" (UID: \"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388\") "
Nov 01 22:27:33.968120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.968089    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume" pod "b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" (UID: "b5b21d62-8ea5-4d4c-add3-7ffa25eb7388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:33.968465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.968439    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:33.970123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:33.970090    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m" pod "b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" (UID: "b5b21d62-8ea5-4d4c-add3-7ffa25eb7388"). InnerVolumeSpecName "kube-api-access-b6m5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:34.069949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.069920    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6m5m\" (UniqueName: \"kubernetes.io/projected/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388-kube-api-access-b6m5m\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:34.367339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.367307    5845 generic.go:356] "Generic (PLEG): container finished" podID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" containerID="0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b" exitCode=0
Nov 01 22:27:34.367498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.367348    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"
Nov 01 22:27:34.367498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.367385    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" event={"ID":"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388","Type":"ContainerDied","Data":"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"}
Nov 01 22:27:34.367498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.367446    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl" event={"ID":"b5b21d62-8ea5-4d4c-add3-7ffa25eb7388","Type":"ContainerDied","Data":"c5a45cfcae58bdf6eb3150e0fd83ffa4428dd9f94a93b6f81d6399aac575be11"}
Nov 01 22:27:34.367498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.367477    5845 scope.go:122] "RemoveContainer" containerID="0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"
Nov 01 22:27:34.378916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.378893    5845 scope.go:122] "RemoveContainer" containerID="0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"
Nov 01 22:27:34.379209 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:34.379181    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b\": container with ID starting with 0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b not found: ID does not exist" containerID="0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"
Nov 01 22:27:34.379345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.379216    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b"} err="failed to get container status \"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b\": rpc error: code = NotFound desc = could not find container \"0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b\": container with ID starting with 0dbdb8e40e026c5635c191a0c676b7b368189e96516cf6684463bd56a700bc5b not found: ID does not exist"
Nov 01 22:27:34.489155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.489101    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:27:34.493007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.492974    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-6b677f4848-bbhnl"]
Nov 01 22:27:34.817502 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:34.817413    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:27:34.863675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:34.863640    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" path="/var/lib/kubelet/pods/b5b21d62-8ea5-4d4c-add3-7ffa25eb7388/volumes"
Nov 01 22:27:35.041679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.041641    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:27:35.041923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.041891    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api" containerID="cri-o://6f42d489349d16459213e6970e94c0532aaa8675fc198760380c1e45d0c88a02" gracePeriod=30
Nov 01 22:27:35.280372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.277955    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:27:35.379658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.379562    5845 generic.go:356] "Generic (PLEG): container finished" podID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerID="6f42d489349d16459213e6970e94c0532aaa8675fc198760380c1e45d0c88a02" exitCode=0
Nov 01 22:27:35.380987 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.380947    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" event={"ID":"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373","Type":"ContainerDied","Data":"6f42d489349d16459213e6970e94c0532aaa8675fc198760380c1e45d0c88a02"}
Nov 01 22:27:35.533844 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.533808    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:27:35.605371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.604834    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\") pod \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") "
Nov 01 22:27:35.605371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.604933    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\") pod \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") "
Nov 01 22:27:35.605371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.604974    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\" (UniqueName: \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\") pod \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") "
Nov 01 22:27:35.605371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.605198    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\") pod \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\" (UID: \"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373\") "
Nov 01 22:27:35.606373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.606345    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca" pod "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" (UID: "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:35.612903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.611532    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8" pod "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" (UID: "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373"). InnerVolumeSpecName "kube-api-access-llts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:35.615009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.614969    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls" pod "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" (UID: "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:27:35.616963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.616942    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs" pod "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" (UID: "aa7ddbf6-b142-4cf1-bb63-4a1ecf531373"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:27:35.706771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.706733    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:35.706771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.706764    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:35.706771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.706779    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:35.707011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:35.706792    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llts8\" (UniqueName: \"kubernetes.io/projected/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373-kube-api-access-llts8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:36.390478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.390444    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"
Nov 01 22:27:36.390883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.390471    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz" event={"ID":"aa7ddbf6-b142-4cf1-bb63-4a1ecf531373","Type":"ContainerDied","Data":"712c8d13a929647a9363cc3ae3d3d53f356c5269bbc267f84209f274b6d64257"}
Nov 01 22:27:36.390883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.390535    5845 scope.go:122] "RemoveContainer" containerID="6f42d489349d16459213e6970e94c0532aaa8675fc198760380c1e45d0c88a02"
Nov 01 22:27:36.415564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.415536    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:27:36.421822 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.421800    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-702002-6bb947cf45-wpzpz"]
Nov 01 22:27:36.864245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:36.864211    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" path="/var/lib/kubelet/pods/aa7ddbf6-b142-4cf1-bb63-4a1ecf531373/volumes"
Nov 01 22:27:37.167535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:37.167456    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:27:37.167766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:37.167737    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy" containerID="cri-o://80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd" gracePeriod=30
Nov 01 22:27:39.189602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.189570    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:27:39.190327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190309    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" containerName="payload-processing"
Nov 01 22:27:39.190378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190328    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" containerName="payload-processing"
Nov 01 22:27:39.190378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190356    5845 cpu_manager.go:414] "containerMap: removing container" podUID="8edc5e52-e102-496d-a562-8278e9bc2354" containerName="payload-processing"
Nov 01 22:27:39.190378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190362    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc5e52-e102-496d-a562-8278e9bc2354" containerName="payload-processing"
Nov 01 22:27:39.190378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190369    5845 cpu_manager.go:414] "containerMap: removing container" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api"
Nov 01 22:27:39.190378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190373    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api"
Nov 01 22:27:39.190540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190397    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:39.190540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190402    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e13660-8961-418b-afd7-1bdf2ee53d34" containerName="istio-proxy"
Nov 01 22:27:39.190540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190521    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc5e52-e102-496d-a562-8278e9bc2354" containerName="payload-processing"
Nov 01 22:27:39.190540 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190529    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7ddbf6-b142-4cf1-bb63-4a1ecf531373" containerName="maas-api"
Nov 01 22:27:39.191561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.190545    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b21d62-8ea5-4d4c-add3-7ffa25eb7388" containerName="payload-processing"
Nov 01 22:27:39.195343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.195095    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.204816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.204561    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a68ac\""
Nov 01 22:27:39.207404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.207316    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:27:39.212106 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.212079    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:27:39.216912 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.216889    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.229693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.226691    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:27:39.229693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.228713    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a68ac-dockercfg-f6wkp\""
Nov 01 22:27:39.280904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.280880    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tgf\" (UniqueName: \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.281047 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.280913    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.281047 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.281014    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpqc\" (UniqueName: \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.281126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.281087    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.391316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.390527    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tgf\" (UniqueName: \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.391316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.390582    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.391316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.390661    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpqc\" (UniqueName: \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.391316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.390780    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.400324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.391912    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.400324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.393133    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.412657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.408131    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tgf\" (UniqueName: \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\") pod \"payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.446684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.446010    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpqc\" (UniqueName: \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\") pod \"payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.515567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.515530    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:39.542897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.542853    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:39.690684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.690628    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:27:39.876857 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.876825    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:27:39.877533 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:39.877511    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63df327d_8632_4a21_b0e9_9c4127eaa5e6.slice/crio-6532ba81961582ac9482d6d329b081d016440aea7fe72a09248c3f7a3f812c1f WatchSource:0}: Error finding container 6532ba81961582ac9482d6d329b081d016440aea7fe72a09248c3f7a3f812c1f: Status 404 returned error can't find the container with id 6532ba81961582ac9482d6d329b081d016440aea7fe72a09248c3f7a3f812c1f
Nov 01 22:27:39.898518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:39.898497    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:27:39.899595 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:39.899567    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c27a609_e302_48a1_8988_fb7af5ec4c39.slice/crio-1864bdc3024aae4eba01c897e8dce9b8cfd45f16b03fc2423aa9acbb80b587c1 WatchSource:0}: Error finding container 1864bdc3024aae4eba01c897e8dce9b8cfd45f16b03fc2423aa9acbb80b587c1: Status 404 returned error can't find the container with id 1864bdc3024aae4eba01c897e8dce9b8cfd45f16b03fc2423aa9acbb80b587c1
Nov 01 22:27:40.520969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.520921    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" event={"ID":"63df327d-8632-4a21-b0e9-9c4127eaa5e6","Type":"ContainerStarted","Data":"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"}
Nov 01 22:27:40.521426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.520993    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" event={"ID":"63df327d-8632-4a21-b0e9-9c4127eaa5e6","Type":"ContainerStarted","Data":"6532ba81961582ac9482d6d329b081d016440aea7fe72a09248c3f7a3f812c1f"}
Nov 01 22:27:40.521426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.521038    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:40.522996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.522966    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" event={"ID":"1c27a609-e302-48a1-8988-fb7af5ec4c39","Type":"ContainerStarted","Data":"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"}
Nov 01 22:27:40.523119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.523013    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:40.523119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.523029    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" event={"ID":"1c27a609-e302-48a1-8988-fb7af5ec4c39","Type":"ContainerStarted","Data":"1864bdc3024aae4eba01c897e8dce9b8cfd45f16b03fc2423aa9acbb80b587c1"}
Nov 01 22:27:40.541620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.541573    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" podStartSLOduration=1.5415599009999998 podStartE2EDuration="1.541559901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:40.539346864 +0000 UTC m=+2563.795526628" watchObservedRunningTime="2026-11-01 22:27:40.541559901 +0000 UTC m=+2563.797739663"
Nov 01 22:27:40.563066 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:40.563018    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" podStartSLOduration=1.563002059 podStartE2EDuration="1.563002059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:40.560587559 +0000 UTC m=+2563.816767336" watchObservedRunningTime="2026-11-01 22:27:40.563002059 +0000 UTC m=+2563.819181821"
Nov 01 22:27:42.499993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.499955    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:27:42.547374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.547333    5845 generic.go:356] "Generic (PLEG): container finished" podID="9871b3c6-0309-406a-b02b-8696a0daf755" containerID="80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd" exitCode=0
Nov 01 22:27:42.547555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.547388    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"
Nov 01 22:27:42.547624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.547556    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerDied","Data":"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"}
Nov 01 22:27:42.547624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.547607    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6" event={"ID":"9871b3c6-0309-406a-b02b-8696a0daf755","Type":"ContainerDied","Data":"d348d20d348c842e8027a83e1eeb07b24a106d78d438896d1207738bcb5a5f81"}
Nov 01 22:27:42.547749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.547639    5845 scope.go:122] "RemoveContainer" containerID="80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"
Nov 01 22:27:42.562556 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.562531    5845 scope.go:122] "RemoveContainer" containerID="cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"
Nov 01 22:27:42.564954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.564931    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.564996    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565045    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565216 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565152    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565216 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565202    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565229    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565279    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565324    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.565429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565416    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\") pod \"9871b3c6-0309-406a-b02b-8696a0daf755\" (UID: \"9871b3c6-0309-406a-b02b-8696a0daf755\") "
Nov 01 22:27:42.566180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565225    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:42.566180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.565771    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:42.566180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.566141    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.566180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.566161    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.566708 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.566664    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:42.567138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.567117    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:42.567238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.567200    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:42.567564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.567538    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:27:42.568243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.568220    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "kube-api-access-57mrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:42.569284 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.569258    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:42.569712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.569694    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token" pod "9871b3c6-0309-406a-b02b-8696a0daf755" (UID: "9871b3c6-0309-406a-b02b-8696a0daf755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:42.610291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.610269    5845 scope.go:122] "RemoveContainer" containerID="80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"
Nov 01 22:27:42.610610 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:42.610590    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd\": container with ID starting with 80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd not found: ID does not exist" containerID="80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"
Nov 01 22:27:42.610662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.610618    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd"} err="failed to get container status \"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd\": rpc error: code = NotFound desc = could not find container \"80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd\": container with ID starting with 80b8d5f40e4fc3a94aaadaf9373e08d3c47364ad74dd899a308cf70b75ac0acd not found: ID does not exist"
Nov 01 22:27:42.610662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.610635    5845 scope.go:122] "RemoveContainer" containerID="cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"
Nov 01 22:27:42.610897 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:42.610880    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf\": container with ID starting with cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf not found: ID does not exist" containerID="cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"
Nov 01 22:27:42.610948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.610905    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf"} err="failed to get container status \"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf\": rpc error: code = NotFound desc = could not find container \"cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf\": container with ID starting with cf54dcda743968f1ee9815d53c1cbb4223a0fbd488425e401b0f91edc13da5cf not found: ID does not exist"
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669717    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669756    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669769    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9871b3c6-0309-406a-b02b-8696a0daf755-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669785    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669799    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9871b3c6-0309-406a-b02b-8696a0daf755-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669813    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mrp\" (UniqueName: \"kubernetes.io/projected/9871b3c6-0309-406a-b02b-8696a0daf755-kube-api-access-57mrp\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.672322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.669830    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9871b3c6-0309-406a-b02b-8696a0daf755-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:42.875852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.875818    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:27:42.876061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.876032    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy" containerID="cri-o://e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d" gracePeriod=30
Nov 01 22:27:42.885265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.885239    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:27:42.893980 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:42.893957    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d6e25f-openshift-default-5b4b4fdb47-85wd6"]
Nov 01 22:27:44.863393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:44.863364    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" path="/var/lib/kubelet/pods/9871b3c6-0309-406a-b02b-8696a0daf755/volumes"
Nov 01 22:27:45.916635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.916602    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:27:45.917414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.917397    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.917414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.917414    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.917518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.917456    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.917518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.917462    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.917580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.917554    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.924947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.924917    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:45.930972 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.930944    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-56452a\""
Nov 01 22:27:45.936514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.936484    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:27:45.943727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.943702    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:27:45.944551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.944535    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9871b3c6-0309-406a-b02b-8696a0daf755" containerName="istio-proxy"
Nov 01 22:27:45.948210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.948190    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:45.973725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.973705    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-56452a-dockercfg-m64tf\""
Nov 01 22:27:45.979043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:45.979014    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:27:46.048094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.045722    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtdn\" (UniqueName: \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.048094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.046070    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.048094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.046163    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqmx\" (UniqueName: \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.048094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.046191    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.151322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.150817    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqmx\" (UniqueName: \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.151322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.150868    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.151322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.150942    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtdn\" (UniqueName: \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.151322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.151233    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.160324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.152112    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.176395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.170543    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.192434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.192393    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtdn\" (UniqueName: \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\") pod \"payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.236325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.229977    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqmx\" (UniqueName: \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\") pod \"payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.245753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.245729    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:46.264963 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.264934    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:46.578924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.578897    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:27:46.580028 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:46.579964    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330a8cb0_89af_4ce5_aa73_58e894eef2fc.slice/crio-a6be6934d6895caf718266b2ed9ca4bb4fc01970fca077eb9e64b91ad276e11a WatchSource:0}: Error finding container a6be6934d6895caf718266b2ed9ca4bb4fc01970fca077eb9e64b91ad276e11a: Status 404 returned error can't find the container with id a6be6934d6895caf718266b2ed9ca4bb4fc01970fca077eb9e64b91ad276e11a
Nov 01 22:27:46.581344 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:46.581284    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07fa865_0c48_457d_8dc7_fd430c157021.slice/crio-bb33d37cf577a02b1e36af132e4e4c41d95fcf131f687e611ae310c9417f8cb0 WatchSource:0}: Error finding container bb33d37cf577a02b1e36af132e4e4c41d95fcf131f687e611ae310c9417f8cb0: Status 404 returned error can't find the container with id bb33d37cf577a02b1e36af132e4e4c41d95fcf131f687e611ae310c9417f8cb0
Nov 01 22:27:46.584332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.584308    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:27:46.602800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.602771    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" event={"ID":"330a8cb0-89af-4ce5-aa73-58e894eef2fc","Type":"ContainerStarted","Data":"a6be6934d6895caf718266b2ed9ca4bb4fc01970fca077eb9e64b91ad276e11a"}
Nov 01 22:27:46.604359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:46.604332    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" event={"ID":"e07fa865-0c48-457d-8dc7-fd430c157021","Type":"ContainerStarted","Data":"bb33d37cf577a02b1e36af132e4e4c41d95fcf131f687e611ae310c9417f8cb0"}
Nov 01 22:27:47.615147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.615105    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" event={"ID":"e07fa865-0c48-457d-8dc7-fd430c157021","Type":"ContainerStarted","Data":"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"}
Nov 01 22:27:47.615620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.615211    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:47.617752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.617724    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" event={"ID":"330a8cb0-89af-4ce5-aa73-58e894eef2fc","Type":"ContainerStarted","Data":"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"}
Nov 01 22:27:47.617889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.617771    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:47.636641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.636578    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" podStartSLOduration=2.6365606169999998 podStartE2EDuration="2.636560617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:47.634466803 +0000 UTC m=+2570.890646570" watchObservedRunningTime="2026-11-01 22:27:47.636560617 +0000 UTC m=+2570.892740381"
Nov 01 22:27:47.657244 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:47.657190    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" podStartSLOduration=2.657173353 podStartE2EDuration="2.657173353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:47.653490363 +0000 UTC m=+2570.909670128" watchObservedRunningTime="2026-11-01 22:27:47.657173353 +0000 UTC m=+2570.913353117"
Nov 01 22:27:48.421785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.421757    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:27:48.537340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537280    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537422    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537453    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537473    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537570    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:48.537656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537629    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537676    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537711    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537777    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.537863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537806    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:48.537863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537850    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\") pod \"6a678a55-3332-4a67-921b-3d4bb53750a4\" (UID: \"6a678a55-3332-4a67-921b-3d4bb53750a4\") "
Nov 01 22:27:48.538019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537923    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:48.538019 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.537959    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:48.539068 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.538789    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:48.540434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.540379    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:27:48.540802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.540753    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "kube-api-access-grthv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541083    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541118    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541138    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541153    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541167    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a678a55-3332-4a67-921b-3d4bb53750a4-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541186    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grthv\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-kube-api-access-grthv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.542338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.541201    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a678a55-3332-4a67-921b-3d4bb53750a4-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.549412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.548853    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:48.560577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.560550    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo" pod "6a678a55-3332-4a67-921b-3d4bb53750a4" (UID: "6a678a55-3332-4a67-921b-3d4bb53750a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:27:48.628023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.627950    5845 generic.go:356] "Generic (PLEG): container finished" podID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerID="e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d" exitCode=0
Nov 01 22:27:48.628023 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.628009    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"
Nov 01 22:27:48.628530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.628031    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" event={"ID":"6a678a55-3332-4a67-921b-3d4bb53750a4","Type":"ContainerDied","Data":"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"}
Nov 01 22:27:48.628530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.628113    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj" event={"ID":"6a678a55-3332-4a67-921b-3d4bb53750a4","Type":"ContainerDied","Data":"ac3893aa58d5faf4e477050afc8bee6f58d9713673ebd180acd4e8fbc24dab13"}
Nov 01 22:27:48.628530 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.628140    5845 scope.go:122] "RemoveContainer" containerID="e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"
Nov 01 22:27:48.640766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.640746    5845 scope.go:122] "RemoveContainer" containerID="e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"
Nov 01 22:27:48.641068 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:48.641044    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d\": container with ID starting with e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d not found: ID does not exist" containerID="e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"
Nov 01 22:27:48.641182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.641080    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d"} err="failed to get container status \"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d\": rpc error: code = NotFound desc = could not find container \"e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d\": container with ID starting with e2efd8de8b94e9f74e77ae2f37570c72742a66090a3ea4e36d5489a0127bdf3d not found: ID does not exist"
Nov 01 22:27:48.642124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.642102    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.642124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.642125    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a678a55-3332-4a67-921b-3d4bb53750a4-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:48.670188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.670161    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:27:48.676955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.676927    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-702002-openshift-default-5588c5c5d9-ktqqj"]
Nov 01 22:27:48.864452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:48.864419    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" path="/var/lib/kubelet/pods/6a678a55-3332-4a67-921b-3d4bb53750a4/volumes"
Nov 01 22:27:49.691713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:49.691666    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:27:50.139246 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.139220    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:27:50.275581 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.275547    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\") pod \"51fe8380-5468-40d1-b793-28069e2d90ab\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") "
Nov 01 22:27:50.275772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.275699    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\" (UniqueName: \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\") pod \"51fe8380-5468-40d1-b793-28069e2d90ab\" (UID: \"51fe8380-5468-40d1-b793-28069e2d90ab\") "
Nov 01 22:27:50.276028 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.276000    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume" pod "51fe8380-5468-40d1-b793-28069e2d90ab" (UID: "51fe8380-5468-40d1-b793-28069e2d90ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:50.276402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.276385    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fe8380-5468-40d1-b793-28069e2d90ab-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:50.278670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.278644    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d" pod "51fe8380-5468-40d1-b793-28069e2d90ab" (UID: "51fe8380-5468-40d1-b793-28069e2d90ab"). InnerVolumeSpecName "kube-api-access-l9r5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:50.378457 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.378413    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9r5d\" (UniqueName: \"kubernetes.io/projected/51fe8380-5468-40d1-b793-28069e2d90ab-kube-api-access-l9r5d\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:50.648547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.648470    5845 generic.go:356] "Generic (PLEG): container finished" podID="51fe8380-5468-40d1-b793-28069e2d90ab" containerID="0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224" exitCode=0
Nov 01 22:27:50.648547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.648513    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"
Nov 01 22:27:50.648717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.648554    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" event={"ID":"51fe8380-5468-40d1-b793-28069e2d90ab","Type":"ContainerDied","Data":"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"}
Nov 01 22:27:50.648717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.648619    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2" event={"ID":"51fe8380-5468-40d1-b793-28069e2d90ab","Type":"ContainerDied","Data":"8534dd09c16743c41cd40bd6876b6d85abb1356b8380a45c81124fcb186ecb3e"}
Nov 01 22:27:50.648717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.648649    5845 scope.go:122] "RemoveContainer" containerID="0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"
Nov 01 22:27:50.661648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.661631    5845 scope.go:122] "RemoveContainer" containerID="0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"
Nov 01 22:27:50.661913 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:27:50.661892    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224\": container with ID starting with 0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224 not found: ID does not exist" containerID="0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"
Nov 01 22:27:50.661964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.661921    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224"} err="failed to get container status \"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224\": rpc error: code = NotFound desc = could not find container \"0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224\": container with ID starting with 0554ce56dc75f8428c7c105ba92688afeb35a5641bd44a60e9ff2b375bda7224 not found: ID does not exist"
Nov 01 22:27:50.673638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.673612    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:27:50.677278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.677256    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-702002-594459b8b7-4rjc2"]
Nov 01 22:27:50.863901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:50.863867    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" path="/var/lib/kubelet/pods/51fe8380-5468-40d1-b793-28069e2d90ab/volumes"
Nov 01 22:27:51.532469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:51.532440    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:27:51.532724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:51.532515    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:27:53.070827 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.070795    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:27:53.071970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.071946    5845 cpu_manager.go:414] "containerMap: removing container" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing"
Nov 01 22:27:53.072059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.071973    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing"
Nov 01 22:27:53.072059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.072032    5845 cpu_manager.go:414] "containerMap: removing container" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy"
Nov 01 22:27:53.072059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.072044    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy"
Nov 01 22:27:53.072304 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.072273    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="51fe8380-5468-40d1-b793-28069e2d90ab" containerName="payload-pre-processing"
Nov 01 22:27:53.072422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.072320    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a678a55-3332-4a67-921b-3d4bb53750a4" containerName="istio-proxy"
Nov 01 22:27:53.078140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.078119    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.093911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.093888    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:27:53.104931 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.104907    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:27:53.109565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.109547    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.121869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.121826    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-e2bf63-openshift-default-dockercfg-tjbqw\""
Nov 01 22:27:53.134737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.134711    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:27:53.250070 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250035    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250107    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250190    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250229    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250260    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250431    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250484    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250569    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250678    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250727    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250774    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250809    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzgt\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250854    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.250969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250927    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.250969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250948    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.251083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.250969    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.251083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.251008    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.251240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.251156    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ml88\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.352942    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.352982    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353012    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353041    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzgt\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353081    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353115    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353143    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353192    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353329 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353217    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353386    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353394    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ml88\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353443    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353544    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353614    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.353678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353678    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353722    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353766    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353814    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353852    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353812    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354017 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353914    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354016    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.353894    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354055    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354130    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354172    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.354396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354204    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.355108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.354490    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.355888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.355860    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.356408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.356388    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.356501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.356391    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.356646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.356629    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.366134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.366106    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.366535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.366515    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzgt\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.366597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.366537    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\") pod \"e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") " pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.366668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.366651    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ml88\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\") pod \"e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") " pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.397671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.397647    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:53.436241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.436213    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:53.463480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.463419    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:27:53.481882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.481559    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:27:53.697807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.697774    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:27:53.698329 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:53.698289    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af9a875_0b45_4e56_bb74_9ea2ff7f53a8.slice/crio-e0085e012439336f70c0680e1891cdf07fb68ef35d84f8a4f213e26a0928da4d WatchSource:0}: Error finding container e0085e012439336f70c0680e1891cdf07fb68ef35d84f8a4f213e26a0928da4d: Status 404 returned error can't find the container with id e0085e012439336f70c0680e1891cdf07fb68ef35d84f8a4f213e26a0928da4d
Nov 01 22:27:53.701400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.701369    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:53.701484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.701434    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:53.701484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.701459    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:53.770800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.770452    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:27:53.801774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.796138    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:53.801774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.796234    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:53.801774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:53.796274    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:27:54.702668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.702621    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" event={"ID":"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8","Type":"ContainerStarted","Data":"75d7b544b7ed05be4a214de4587ba8a0bf12eda1b04faa8320f26d8b8b8821de"}
Nov 01 22:27:54.703158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.702699    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" event={"ID":"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8","Type":"ContainerStarted","Data":"e0085e012439336f70c0680e1891cdf07fb68ef35d84f8a4f213e26a0928da4d"}
Nov 01 22:27:54.705099 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.705069    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" event={"ID":"46d4bcd7-f7ed-4f73-95a0-235581345bec","Type":"ContainerStarted","Data":"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"}
Nov 01 22:27:54.705228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.705115    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" event={"ID":"46d4bcd7-f7ed-4f73-95a0-235581345bec","Type":"ContainerStarted","Data":"ac802febe9d9854a8576456b2a6ee0ef855b08fb1071e389edaaabb7e2cea3d3"}
Nov 01 22:27:54.728529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.728478    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podStartSLOduration=1.7284631689999999 podStartE2EDuration="1.728463169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.726023857 +0000 UTC m=+2577.982203621" watchObservedRunningTime="2026-11-01 22:27:54.728463169 +0000 UTC m=+2577.984642933"
Nov 01 22:27:54.769323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:54.768819    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podStartSLOduration=1.768794796 podStartE2EDuration="1.768794796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:27:54.755541965 +0000 UTC m=+2578.011721728" watchObservedRunningTime="2026-11-01 22:27:54.768794796 +0000 UTC m=+2578.024974560"
Nov 01 22:27:55.401419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.401383    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:27:55.434503 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.434464    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:27:55.435404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.435364    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:27:55.439320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.436666    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:27:55.459321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.454482    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:27:55.459321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:55.454571    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:27:56.418320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:56.417610    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:27:56.418320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:56.417675    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:27:56.437735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:56.437631    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:27:56.437735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:56.437721    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:27:57.398729 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:57.398691    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:27:57.398938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:57.398742    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:27:57.437543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:57.437513    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:27:57.437896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:57.437558    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:27:58.398934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.398798    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:27:58.398934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.398868    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:27:58.437501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.437456    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:27:58.437691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.437517    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:27:58.473416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.473364    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing" containerID="cri-o://656f91fa997998cf54bab9346f8035a57b639539af77e9708d8566be8751e025" gracePeriod=25
Nov 01 22:27:58.483600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.483550    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" containerID="cri-o://a8143bbff9469610b961b0ebe2c0281df234b4caaa8a970d314b4b458cfa3446" gracePeriod=25
Nov 01 22:27:58.528389 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:58.527869    5845 logging.go:55] [core] [Channel #7473 SubChannel #7474]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.105:50052: connect: connection refused"
Nov 01 22:27:58.528389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.528127    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:27:58.631135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.630982    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:27:58.631135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.631060    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:27:58.813638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.812834    5845 generic.go:356] "Generic (PLEG): container finished" podID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerID="656f91fa997998cf54bab9346f8035a57b639539af77e9708d8566be8751e025" exitCode=0
Nov 01 22:27:58.813638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:58.813155    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" event={"ID":"71d002b5-67d0-4d80-a9c5-4b6f72d14837","Type":"ContainerDied","Data":"656f91fa997998cf54bab9346f8035a57b639539af77e9708d8566be8751e025"}
Nov 01 22:27:59.236235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.236211    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:27:59.339155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.339045    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\" (UniqueName: \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\") pod \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") "
Nov 01 22:27:59.339371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.339345    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\") pod \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\" (UID: \"71d002b5-67d0-4d80-a9c5-4b6f72d14837\") "
Nov 01 22:27:59.340786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.340743    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume" pod "71d002b5-67d0-4d80-a9c5-4b6f72d14837" (UID: "71d002b5-67d0-4d80-a9c5-4b6f72d14837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:27:59.343500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.343468    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s" pod "71d002b5-67d0-4d80-a9c5-4b6f72d14837" (UID: "71d002b5-67d0-4d80-a9c5-4b6f72d14837"). InnerVolumeSpecName "kube-api-access-qgq9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:27:59.403322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.402474    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:27:59.403322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.402533    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:27:59.437236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.437206    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:27:59.437402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.437256    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:27:59.441757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.441731    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71d002b5-67d0-4d80-a9c5-4b6f72d14837-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:59.441757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.441758    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgq9s\" (UniqueName: \"kubernetes.io/projected/71d002b5-67d0-4d80-a9c5-4b6f72d14837-kube-api-access-qgq9s\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:27:59.528455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.528411    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.105:50052\" within 1s: context deadline exceeded"
Nov 01 22:27:59.528788 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:27:59.528767    5845 logging.go:55] [core] [Channel #7473 SubChannel #7474]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.105:50052", ServerName: "10.134.0.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.105:50052: operation was canceled"
Nov 01 22:27:59.825510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.825476    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs" event={"ID":"71d002b5-67d0-4d80-a9c5-4b6f72d14837","Type":"ContainerDied","Data":"08589b67f74436893c44c99889177ea1ae756ba070f792d9058d70c4e914bce3"}
Nov 01 22:27:59.825510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.825501    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"
Nov 01 22:27:59.825790 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.825548    5845 scope.go:122] "RemoveContainer" containerID="656f91fa997998cf54bab9346f8035a57b639539af77e9708d8566be8751e025"
Nov 01 22:27:59.908395 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.908352    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:27:59.912029 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:27:59.912001    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-de0cba-7dc764dbd4-glwvs"]
Nov 01 22:28:00.236555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.235204    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:28:00.236555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.236075    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy" containerID="cri-o://cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b" gracePeriod=30
Nov 01 22:28:00.427774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.427738    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 01 22:28:00.428204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.428097    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 01 22:28:00.448319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.445865    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 01 22:28:00.448319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.445914    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 01 22:28:00.873321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:00.873054    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" path="/var/lib/kubelet/pods/71d002b5-67d0-4d80-a9c5-4b6f72d14837/volumes"
Nov 01 22:28:01.174501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.174420    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:01.175471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.175445    5845 cpu_manager.go:414] "containerMap: removing container" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing"
Nov 01 22:28:01.175604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.175475    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing"
Nov 01 22:28:01.175734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.175719    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d002b5-67d0-4d80-a9c5-4b6f72d14837" containerName="payload-processing"
Nov 01 22:28:01.180983 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.180962    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.185567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.185218    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db715b-dockercfg-mlmn9\""
Nov 01 22:28:01.188728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.187350    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:01.265202 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.265168    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db715b\""
Nov 01 22:28:01.351809 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.351382    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k676z\" (UniqueName: \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.351809 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.351500    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.460467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.460386    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:28:01.462524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.462501    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:28:01.463806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.463781    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k676z\" (UniqueName: \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.492696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.492669    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.516345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.493790    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.516345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.504477    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k676z\" (UniqueName: \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\") pod \"payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.516345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.473112    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:28:01.517758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.517732    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:01.572723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.558634    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:28:01.572723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.563636    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:28:01.572723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.572458    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:28:01.933534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:01.933503    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:01.934356 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:28:01.934329    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6e92fd_b731_481a_a0ef_bb81369253be.slice/crio-2b77aaf4649c280e0d3d28536bf66136e20e5e1586e2381591d6ecc43469d204 WatchSource:0}: Error finding container 2b77aaf4649c280e0d3d28536bf66136e20e5e1586e2381591d6ecc43469d204: Status 404 returned error can't find the container with id 2b77aaf4649c280e0d3d28536bf66136e20e5e1586e2381591d6ecc43469d204
Nov 01 22:28:02.863119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:02.863084    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" event={"ID":"7c6e92fd-b731-481a-a0ef-bb81369253be","Type":"ContainerStarted","Data":"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"}
Nov 01 22:28:02.863547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:02.863140    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" event={"ID":"7c6e92fd-b731-481a-a0ef-bb81369253be","Type":"ContainerStarted","Data":"2b77aaf4649c280e0d3d28536bf66136e20e5e1586e2381591d6ecc43469d204"}
Nov 01 22:28:02.863547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:02.863169    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:02.899134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:02.880560    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" podStartSLOduration=1.880542741 podStartE2EDuration="1.880542741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:02.877033486 +0000 UTC m=+2586.133213249" watchObservedRunningTime="2026-11-01 22:28:02.880542741 +0000 UTC m=+2586.136722506"
Nov 01 22:28:04.819251 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:04.819213    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:28:05.546013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.545988    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:28:05.685349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.685313    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 22:28:05.695784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.695748    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.730881    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.730982    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731057    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731094    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731122    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731221    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731792    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731857    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.731932    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\") pod \"fce9762c-ab3f-434e-ad01-db18977833d3\" (UID: \"fce9762c-ab3f-434e-ad01-db18977833d3\") "
Nov 01 22:28:05.733787 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.733758    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:05.735142 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.735112    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:05.735697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.735663    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:05.735871 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.735842    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:05.736263 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.736243    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:05.738539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.738509    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:05.739788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.739760    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "kube-api-access-zswc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:05.746921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.746892    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:28:05.748337 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.748270    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token" pod "fce9762c-ab3f-434e-ad01-db18977833d3" (UID: "fce9762c-ab3f-434e-ad01-db18977833d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:05.839416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839385    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce9762c-ab3f-434e-ad01-db18977833d3-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.839416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839411    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.839416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839421    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839430    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce9762c-ab3f-434e-ad01-db18977833d3-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839438    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswc7\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-kube-api-access-zswc7\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839448    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839456    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce9762c-ab3f-434e-ad01-db18977833d3-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839464    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.840026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.839473    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce9762c-ab3f-434e-ad01-db18977833d3-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:05.931913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.931878    5845 generic.go:356] "Generic (PLEG): container finished" podID="fce9762c-ab3f-434e-ad01-db18977833d3" containerID="cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b" exitCode=0
Nov 01 22:28:05.932077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.931934    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"
Nov 01 22:28:05.932077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.931962    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" event={"ID":"fce9762c-ab3f-434e-ad01-db18977833d3","Type":"ContainerDied","Data":"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"}
Nov 01 22:28:05.932077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.932013    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s" event={"ID":"fce9762c-ab3f-434e-ad01-db18977833d3","Type":"ContainerDied","Data":"921a0a5af42c0ec40a98332d593ac7a03ddd8a33ea89c15783ebf952c02a4e83"}
Nov 01 22:28:05.932077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.932033    5845 scope.go:122] "RemoveContainer" containerID="cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"
Nov 01 22:28:05.943705 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.943683    5845 scope.go:122] "RemoveContainer" containerID="cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"
Nov 01 22:28:05.943969 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:05.943949    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b\": container with ID starting with cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b not found: ID does not exist" containerID="cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"
Nov 01 22:28:05.944024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.943977    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b"} err="failed to get container status \"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b\": rpc error: code = NotFound desc = could not find container \"cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b\": container with ID starting with cbfff1f7a7fd2645a92153954a182b4f07d5c4fe3c92274694c85aa382e1e96b not found: ID does not exist"
Nov 01 22:28:05.958746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.958720    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:28:05.962385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:05.962362    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-da34ca-openshift-default-6db76b45c-jsm5s"]
Nov 01 22:28:06.890495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:06.890451    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" path="/var/lib/kubelet/pods/fce9762c-ab3f-434e-ad01-db18977833d3/volumes"
Nov 01 22:28:08.527137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:08.527094    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:28:13.528831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:13.528797    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:28:13.529214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:13.529003    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy" containerID="cri-o://8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8" gracePeriod=30
Nov 01 22:28:13.920481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:13.920404    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:15.154446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:15.154409    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:28:15.155112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:15.154468    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:28:17.443090 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.443062    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:28:17.563288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.563249    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.563526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.563342    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.563645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.563615    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:17.563821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.563801    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.563908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.563875    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.564500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.564466    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.564807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.564766    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:17.564934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.564917    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.565159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.565140    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:17.565241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.565189    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.565582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.565560    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:17.565689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.565622    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.566057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.566038    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\") pod \"637a4546-919b-448a-80da-944af0aaf4b3\" (UID: \"637a4546-919b-448a-80da-944af0aaf4b3\") "
Nov 01 22:28:17.566556 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.566526    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:17.567169 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.567147    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.567746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.567246    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.567746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.567262    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.567746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.567276    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.567746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.567290    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637a4546-919b-448a-80da-944af0aaf4b3-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.568113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.568093    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "kube-api-access-tm75l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:17.568439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.568420    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:28:17.570450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.570419    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:17.571507 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.571482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token" pod "637a4546-919b-448a-80da-944af0aaf4b3" (UID: "637a4546-919b-448a-80da-944af0aaf4b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:17.669113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.669011    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm75l\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-kube-api-access-tm75l\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.669113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.669054    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637a4546-919b-448a-80da-944af0aaf4b3-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.669113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.669072    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637a4546-919b-448a-80da-944af0aaf4b3-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:17.669113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:17.669087    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637a4546-919b-448a-80da-944af0aaf4b3-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:18.098825 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.098792    5845 generic.go:356] "Generic (PLEG): container finished" podID="637a4546-919b-448a-80da-944af0aaf4b3" containerID="8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8" exitCode=0
Nov 01 22:28:18.099033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.098839    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"
Nov 01 22:28:18.099033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.098871    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" event={"ID":"637a4546-919b-448a-80da-944af0aaf4b3","Type":"ContainerDied","Data":"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"}
Nov 01 22:28:18.099033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.098931    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq" event={"ID":"637a4546-919b-448a-80da-944af0aaf4b3","Type":"ContainerDied","Data":"369262d1f115b28765baa46c162ef212c4269b25a48a9db48a781efb48ebd108"}
Nov 01 22:28:18.099033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.098958    5845 scope.go:122] "RemoveContainer" containerID="8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"
Nov 01 22:28:18.112739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.112718    5845 scope.go:122] "RemoveContainer" containerID="8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"
Nov 01 22:28:18.113038 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:18.113019    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8\": container with ID starting with 8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8 not found: ID does not exist" containerID="8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"
Nov 01 22:28:18.113080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.113047    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8"} err="failed to get container status \"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8\": rpc error: code = NotFound desc = could not find container \"8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8\": container with ID starting with 8c7983b0e3263108df6135bb3b7ed90dc2257f5f7ba7f34700263c3419d77bc8 not found: ID does not exist"
Nov 01 22:28:18.127006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.126976    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:28:18.134739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.134713    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0bb375-openshift-default-6689c9665-mjzkq"]
Nov 01 22:28:18.538343 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:28:18.535896    5845 logging.go:55] [core] [Channel #7535 SubChannel #7536]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.106:50052", ServerName: "10.134.0.106:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.106:50052: connect: connection refused"
Nov 01 22:28:18.909386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:18.905314    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="637a4546-919b-448a-80da-944af0aaf4b3" path="/var/lib/kubelet/pods/637a4546-919b-448a-80da-944af0aaf4b3/volumes"
Nov 01 22:28:19.232091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.231905    5845 generic.go:356] "Generic (PLEG): container finished" podID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerID="a8143bbff9469610b961b0ebe2c0281df234b4caaa8a970d314b4b458cfa3446" exitCode=0
Nov 01 22:28:19.232600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.232569    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" event={"ID":"468b5cf4-14a0-4954-8c5b-870be68ecf1e","Type":"ContainerDied","Data":"a8143bbff9469610b961b0ebe2c0281df234b4caaa8a970d314b4b458cfa3446"}
Nov 01 22:28:19.280474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.280437    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:28:19.389377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.388712    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\" (UniqueName: \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\") pod \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") "
Nov 01 22:28:19.389377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.388852    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\") pod \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\" (UID: \"468b5cf4-14a0-4954-8c5b-870be68ecf1e\") "
Nov 01 22:28:19.391618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.391587    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume" pod "468b5cf4-14a0-4954-8c5b-870be68ecf1e" (UID: "468b5cf4-14a0-4954-8c5b-870be68ecf1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:19.392893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.392864    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt" pod "468b5cf4-14a0-4954-8c5b-870be68ecf1e" (UID: "468b5cf4-14a0-4954-8c5b-870be68ecf1e"). InnerVolumeSpecName "kube-api-access-jcnmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:19.493624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.493536    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/468b5cf4-14a0-4954-8c5b-870be68ecf1e-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:19.493624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.493572    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnmt\" (UniqueName: \"kubernetes.io/projected/468b5cf4-14a0-4954-8c5b-870be68ecf1e-kube-api-access-jcnmt\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:19.536546 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.536504    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.106:50052\" within 1s: context deadline exceeded"
Nov 01 22:28:19.536717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:19.536589    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:28:20.280306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.278754    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"
Nov 01 22:28:20.280306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.278742    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl" event={"ID":"468b5cf4-14a0-4954-8c5b-870be68ecf1e","Type":"ContainerDied","Data":"ede993efc5dcb8388ba84eb831087f5f5194c4a62ec36368167c06824016bafb"}
Nov 01 22:28:20.280306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.278834    5845 scope.go:122] "RemoveContainer" containerID="a8143bbff9469610b961b0ebe2c0281df234b4caaa8a970d314b4b458cfa3446"
Nov 01 22:28:20.321813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.321776    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:28:20.332735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.332706    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-de0cba-78f5f7494c-fhbbl"]
Nov 01 22:28:20.723256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.723206    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" containerID="cri-o://402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb" gracePeriod=45
Nov 01 22:28:20.725338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.725272    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" containerID="cri-o://c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c" gracePeriod=45
Nov 01 22:28:20.783967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.783930    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused"
Nov 01 22:28:20.887061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:20.887026    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" path="/var/lib/kubelet/pods/468b5cf4-14a0-4954-8c5b-870be68ecf1e/volumes"
Nov 01 22:28:21.073420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.073384    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused"
Nov 01 22:28:21.187852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.187501    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:28:21.244225 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.244190    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263498    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263569    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\" (UniqueName: \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263638    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263728    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263845    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.265080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.263936    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\" (UniqueName: \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\") pod \"b321c879-897d-4097-b678-9b37339df479\" (UID: \"b321c879-897d-4097-b678-9b37339df479\") "
Nov 01 22:28:21.269220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.266438    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.269220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.266900    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.269220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.266975    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.271854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.271783    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.271854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.271812    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.271854 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.271827    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.288011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.287196    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:28:21.288011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.287977    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.291831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.291799    5845 generic.go:356] "Generic (PLEG): container finished" podID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerID="c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c" exitCode=0
Nov 01 22:28:21.292607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.292016    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" event={"ID":"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf","Type":"ContainerDied","Data":"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"}
Nov 01 22:28:21.292607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.292066    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw" event={"ID":"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf","Type":"ContainerDied","Data":"d9c0a63e61cb5b5e7166ce2e246ff937a28a76c5c926880654bc5fc840aa4ad0"}
Nov 01 22:28:21.292607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.292090    5845 scope.go:122] "RemoveContainer" containerID="c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"
Nov 01 22:28:21.292607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.292035    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"
Nov 01 22:28:21.296377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.295649    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k" pod "b321c879-897d-4097-b678-9b37339df479" (UID: "b321c879-897d-4097-b678-9b37339df479"). InnerVolumeSpecName "kube-api-access-fgr8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:21.298868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.298285    5845 generic.go:356] "Generic (PLEG): container finished" podID="b321c879-897d-4097-b678-9b37339df479" containerID="402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb" exitCode=0
Nov 01 22:28:21.298868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.298445    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" event={"ID":"b321c879-897d-4097-b678-9b37339df479","Type":"ContainerDied","Data":"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"}
Nov 01 22:28:21.298868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.298480    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk" event={"ID":"b321c879-897d-4097-b678-9b37339df479","Type":"ContainerDied","Data":"4bf3452700875ff9bd45e0b4ccfbcbacaef3a0a8ef94bc6a86e6da156d958497"}
Nov 01 22:28:21.298868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.298549    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"
Nov 01 22:28:21.334954 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.334619    5845 scope.go:122] "RemoveContainer" containerID="c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"
Nov 01 22:28:21.339394 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:21.339286    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c\": container with ID starting with c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c not found: ID does not exist" containerID="c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"
Nov 01 22:28:21.339394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.339346    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c"} err="failed to get container status \"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c\": rpc error: code = NotFound desc = could not find container \"c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c\": container with ID starting with c99887c1c821eca7482e935c91eac203ab10fb11821a07a07c5114234593a64c not found: ID does not exist"
Nov 01 22:28:21.339394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.339374    5845 scope.go:122] "RemoveContainer" containerID="402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"
Nov 01 22:28:21.395731 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.395688    5845 scope.go:122] "RemoveContainer" containerID="402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"
Nov 01 22:28:21.396397 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:21.396369    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb\": container with ID starting with 402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb not found: ID does not exist" containerID="402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"
Nov 01 22:28:21.396581 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.396551    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb"} err="failed to get container status \"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb\": rpc error: code = NotFound desc = could not find container \"402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb\": container with ID starting with 402545cebb4b2b5419e9ba6570dd3a818450475116b81f344a1caf6468d2d1cb not found: ID does not exist"
Nov 01 22:28:21.594405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.553001    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.615486    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.615663    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.615708    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\" (UniqueName: \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.615757    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.615968    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\") pod \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\" (UID: \"1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf\") "
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.617478    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b321c879-897d-4097-b678-9b37339df479-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.617516    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b321c879-897d-4097-b678-9b37339df479-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.631454 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.617536    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgr8k\" (UniqueName: \"kubernetes.io/projected/b321c879-897d-4097-b678-9b37339df479-kube-api-access-fgr8k\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.633805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.599163    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 22:28:21.676649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.676602    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.676856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.676823    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.700903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.699442    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.700903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.700718    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:21.700903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.700781    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:28:21.702432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.702408    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/distinct-w6-fb9597-kserve-774b85cb69-mkcfk"]
Nov 01 22:28:21.706087 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.705400    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc" pod "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" (UID: "1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf"). InnerVolumeSpecName "kube-api-access-54clc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:21.718709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718680    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.718709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718708    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.718894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718718    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.718894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718727    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.718894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718735    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.718894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.718744    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54clc\" (UniqueName: \"kubernetes.io/projected/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf-kube-api-access-54clc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:21.880745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.880653    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 22:28:21.973838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.973803    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 22:28:21.980237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:21.980206    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fb9597/premium-w6-fb9597-kserve-565984d6-lhnrw"]
Nov 01 22:28:23.005943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:23.004919    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" path="/var/lib/kubelet/pods/1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf/volumes"
Nov 01 22:28:23.005943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:23.005558    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b321c879-897d-4097-b678-9b37339df479" path="/var/lib/kubelet/pods/b321c879-897d-4097-b678-9b37339df479/volumes"
Nov 01 22:28:30.375558 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.375517    5845 generic.go:356] "Generic (PLEG): container finished" podID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerID="ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25" exitCode=0
Nov 01 22:28:30.376009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.375590    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerDied","Data":"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"}
Nov 01 22:28:30.376061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.376027    5845 scope.go:122] "RemoveContainer" containerID="ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"
Nov 01 22:28:30.377845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.377817    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:30.377921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.377885    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:30.377970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:30.377929    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:31.388260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:31.388226    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerStarted","Data":"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"}
Nov 01 22:28:31.663027 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:31.662943    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:28:31.794350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:31.793789    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:28:31.799594 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:31.799557    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:31.799791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:31.799641    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:32.791590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:32.791557    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:32.792006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:32.791616    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:33.792151 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:33.792120    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:33.792580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:33.792169    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:34.792237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:34.792196    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:34.792730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:34.792275    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:34.818390 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:34.818356    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:28:35.791655 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:35.791621    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:35.791853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:35.791688    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:36.792241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:36.792200    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:28:36.792713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:36.792259    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 01 22:28:36.912250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:36.912206    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" containerID="cri-o://a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df" gracePeriod=45
Nov 01 22:28:37.030884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.030848    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 01 22:28:37.255285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.255257    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:28:37.332177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332136    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.332402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332209    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.332402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332316    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.332543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332455    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.332543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332515    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\" (UniqueName: \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.332929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.332788    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\") pod \"ad5419e4-dceb-4777-b863-cddeb99eb425\" (UID: \"ad5419e4-dceb-4777-b863-cddeb99eb425\") "
Nov 01 22:28:37.333255 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.333200    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.333455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.333422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.333761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.333741    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.333914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.333901    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.334061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.333783    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.336181 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.336143    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.336334 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.336237    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "kube-api-access-jf2ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:37.336568 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.336545    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs" pod "ad5419e4-dceb-4777-b863-cddeb99eb425" (UID: "ad5419e4-dceb-4777-b863-cddeb99eb425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:28:37.434597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.434501    5845 generic.go:356] "Generic (PLEG): container finished" podID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerID="a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df" exitCode=0
Nov 01 22:28:37.434597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.434548    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" event={"ID":"ad5419e4-dceb-4777-b863-cddeb99eb425","Type":"ContainerDied","Data":"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"}
Nov 01 22:28:37.434597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.434555    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"
Nov 01 22:28:37.434597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.434580    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w" event={"ID":"ad5419e4-dceb-4777-b863-cddeb99eb425","Type":"ContainerDied","Data":"7860a84fdc4013bdcbb489f8a9269005f166ad1df44247eda85d9f65c64a3027"}
Nov 01 22:28:37.434597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.434596    5845 scope.go:122] "RemoveContainer" containerID="a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"
Nov 01 22:28:37.435440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.435416    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5419e4-dceb-4777-b863-cddeb99eb425-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.435440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.435441    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.435697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.435456    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5419e4-dceb-4777-b863-cddeb99eb425-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.435697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.435470    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf2ks\" (UniqueName: \"kubernetes.io/projected/ad5419e4-dceb-4777-b863-cddeb99eb425-kube-api-access-jf2ks\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.446472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.446450    5845 scope.go:122] "RemoveContainer" containerID="a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"
Nov 01 22:28:37.446732 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:37.446710    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df\": container with ID starting with a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df not found: ID does not exist" containerID="a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"
Nov 01 22:28:37.446782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.446740    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df"} err="failed to get container status \"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df\": rpc error: code = NotFound desc = could not find container \"a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df\": container with ID starting with a3fbf374fc667359bc7dac835d1f7da6d260e235a13a13e9cb90ce832b1bc7df not found: ID does not exist"
Nov 01 22:28:37.462488 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.462465    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 22:28:37.466444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.466424    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f033b1/simulator-w2-f033b1-kserve-557d698566-9kk5w"]
Nov 01 22:28:37.796647 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.796619    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:28:37.797753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:37.797724    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:28:38.862511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:38.862481    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" path="/var/lib/kubelet/pods/ad5419e4-dceb-4777-b863-cddeb99eb425/volumes"
Nov 01 22:28:39.486067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:39.486029    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:41.094775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.094732    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:28:41.095274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.094978    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" podUID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" containerName="payload-processing" containerID="cri-o://caf51b41cf910d5f705653e393d8746e412399a668b7ca434a1c440370619348" gracePeriod=30
Nov 01 22:28:41.475252 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.475175    5845 generic.go:356] "Generic (PLEG): container finished" podID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" containerID="caf51b41cf910d5f705653e393d8746e412399a668b7ca434a1c440370619348" exitCode=0
Nov 01 22:28:41.475513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.475399    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" event={"ID":"e99683f0-28b4-48cd-9b26-6a386ba7d1f2","Type":"ContainerDied","Data":"caf51b41cf910d5f705653e393d8746e412399a668b7ca434a1c440370619348"}
Nov 01 22:28:41.549179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.549141    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:28:41.642837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.642768    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\" (UniqueName: \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\") pod \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") "
Nov 01 22:28:41.643034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.642937    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\") pod \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\" (UID: \"e99683f0-28b4-48cd-9b26-6a386ba7d1f2\") "
Nov 01 22:28:41.643444 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.643400    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume" pod "e99683f0-28b4-48cd-9b26-6a386ba7d1f2" (UID: "e99683f0-28b4-48cd-9b26-6a386ba7d1f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:41.643641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.643621    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:41.646285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.646255    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45" pod "e99683f0-28b4-48cd-9b26-6a386ba7d1f2" (UID: "e99683f0-28b4-48cd-9b26-6a386ba7d1f2"). InnerVolumeSpecName "kube-api-access-n4r45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:41.787410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:41.786759    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4r45\" (UniqueName: \"kubernetes.io/projected/e99683f0-28b4-48cd-9b26-6a386ba7d1f2-kube-api-access-n4r45\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:42.485974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.485937    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m" event={"ID":"e99683f0-28b4-48cd-9b26-6a386ba7d1f2","Type":"ContainerDied","Data":"aaf018287890d8b066f340b0ccf3aa14234fc56cb9ed8dd55fc12e751c7d12bf"}
Nov 01 22:28:42.485974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.485965    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"
Nov 01 22:28:42.486554 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.486015    5845 scope.go:122] "RemoveContainer" containerID="caf51b41cf910d5f705653e393d8746e412399a668b7ca434a1c440370619348"
Nov 01 22:28:42.521851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.521820    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:28:42.526633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.526608    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a00c6a-6b5b58b4cd-24z4m"]
Nov 01 22:28:42.866576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:42.866487    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" path="/var/lib/kubelet/pods/e99683f0-28b4-48cd-9b26-6a386ba7d1f2/volumes"
Nov 01 22:28:44.486702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.486646    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" podUID="7c6e92fd-b731-481a-a0ef-bb81369253be" containerName="payload-pre-processing" containerID="cri-o://c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6" gracePeriod=25
Nov 01 22:28:44.835969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.835941    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:44.975015 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.974982    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\" (UniqueName: \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\") pod \"7c6e92fd-b731-481a-a0ef-bb81369253be\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") "
Nov 01 22:28:44.975356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.975227    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\") pod \"7c6e92fd-b731-481a-a0ef-bb81369253be\" (UID: \"7c6e92fd-b731-481a-a0ef-bb81369253be\") "
Nov 01 22:28:44.975596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.975568    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume" pod "7c6e92fd-b731-481a-a0ef-bb81369253be" (UID: "7c6e92fd-b731-481a-a0ef-bb81369253be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:44.976040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.976008    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c6e92fd-b731-481a-a0ef-bb81369253be-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:44.977498 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:44.977478    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z" pod "7c6e92fd-b731-481a-a0ef-bb81369253be" (UID: "7c6e92fd-b731-481a-a0ef-bb81369253be"). InnerVolumeSpecName "kube-api-access-k676z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:45.077257 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.077194    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k676z\" (UniqueName: \"kubernetes.io/projected/7c6e92fd-b731-481a-a0ef-bb81369253be-kube-api-access-k676z\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:45.513423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.513383    5845 generic.go:356] "Generic (PLEG): container finished" podID="7c6e92fd-b731-481a-a0ef-bb81369253be" containerID="c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6" exitCode=0
Nov 01 22:28:45.513878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.513433    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"
Nov 01 22:28:45.513878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.513467    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" event={"ID":"7c6e92fd-b731-481a-a0ef-bb81369253be","Type":"ContainerDied","Data":"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"}
Nov 01 22:28:45.513878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.513525    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5" event={"ID":"7c6e92fd-b731-481a-a0ef-bb81369253be","Type":"ContainerDied","Data":"2b77aaf4649c280e0d3d28536bf66136e20e5e1586e2381591d6ecc43469d204"}
Nov 01 22:28:45.513878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.513547    5845 scope.go:122] "RemoveContainer" containerID="c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"
Nov 01 22:28:45.527755 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.527723    5845 scope.go:122] "RemoveContainer" containerID="c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"
Nov 01 22:28:45.528035 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:45.528015    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6\": container with ID starting with c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6 not found: ID does not exist" containerID="c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"
Nov 01 22:28:45.528128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.528045    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6"} err="failed to get container status \"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6\": rpc error: code = NotFound desc = could not find container \"c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6\": container with ID starting with c70ee1048087f8bd1e4ef760e3623842433780462391d31db366c04dfae16aa6 not found: ID does not exist"
Nov 01 22:28:45.539760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.539739    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:45.545460 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:45.545438    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db715b-67d594b7d7-fv9b5"]
Nov 01 22:28:46.522619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:46.522540    5845 generic.go:356] "Generic (PLEG): container finished" podID="cfeec369-c249-494b-b2a2-8c0753b3d17e" containerID="5e252301f4bb73c5f044f884e505e8b02bd5de82b66c1dbb906e59c314e11ae4" exitCode=0
Nov 01 22:28:46.523285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:46.522626    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" event={"ID":"cfeec369-c249-494b-b2a2-8c0753b3d17e","Type":"ContainerDied","Data":"5e252301f4bb73c5f044f884e505e8b02bd5de82b66c1dbb906e59c314e11ae4"}
Nov 01 22:28:46.523285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:46.522681    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5" event={"ID":"cfeec369-c249-494b-b2a2-8c0753b3d17e","Type":"ContainerStarted","Data":"c68a15d396d2e2e22372722fda3e10622686cef06a8e9c3e900a5891e23036f1"}
Nov 01 22:28:46.523285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:46.522724    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 22:28:46.863102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:46.863019    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6e92fd-b731-481a-a0ef-bb81369253be" path="/var/lib/kubelet/pods/7c6e92fd-b731-481a-a0ef-bb81369253be/volumes"
Nov 01 22:28:52.796089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:52.796052    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 22:28:52.796582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:52.796366    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api" containerID="cri-o://a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc" gracePeriod=30
Nov 01 22:28:53.145323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.144850    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 22:28:53.229994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.228856    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\") pod \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") "
Nov 01 22:28:53.229994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.228912    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\") pod \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") "
Nov 01 22:28:53.229994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.229166    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\") pod \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") "
Nov 01 22:28:53.229994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.229195    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\" (UniqueName: \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\") pod \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\" (UID: \"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7\") "
Nov 01 22:28:53.236149 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.233862    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca" pod "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" (UID: "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:28:53.291322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.287468    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9" pod "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" (UID: "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7"). InnerVolumeSpecName "kube-api-access-dxhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:53.291322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.288655    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls" pod "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" (UID: "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:28:53.291580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.291454    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs" pod "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" (UID: "0b19f8a9-1528-4a10-8935-afb3a4ef2ee7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:28:53.333779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.331943    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:53.333779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.331984    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:53.333779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.332001    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:53.333779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.332016    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxhc9\" (UniqueName: \"kubernetes.io/projected/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7-kube-api-access-dxhc9\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:53.580093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.580055    5845 generic.go:356] "Generic (PLEG): container finished" podID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerID="a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc" exitCode=0
Nov 01 22:28:53.580322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.580104    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"
Nov 01 22:28:53.580322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.580140    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" event={"ID":"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7","Type":"ContainerDied","Data":"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"}
Nov 01 22:28:53.580322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.580202    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf" event={"ID":"0b19f8a9-1528-4a10-8935-afb3a4ef2ee7","Type":"ContainerDied","Data":"b2e74732e28fc631cbce1e072b0a0efd382bafea305d075701808fc721165a9d"}
Nov 01 22:28:53.580322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.580231    5845 scope.go:122] "RemoveContainer" containerID="a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"
Nov 01 22:28:53.594901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.594787    5845 scope.go:122] "RemoveContainer" containerID="a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"
Nov 01 22:28:53.595117 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:28:53.595090    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc\": container with ID starting with a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc not found: ID does not exist" containerID="a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"
Nov 01 22:28:53.596069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.595128    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc"} err="failed to get container status \"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc\": rpc error: code = NotFound desc = could not find container \"a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc\": container with ID starting with a32eaeaac2116f976fd5d16afb4519d2bb12b128c05755e412ef9b90d8d5f1fc not found: ID does not exist"
Nov 01 22:28:53.608648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.608623    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 22:28:53.621345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:53.621321    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fb9597-6c6dd74ccc-rtmvf"]
Nov 01 22:28:54.862983 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:54.862936    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" path="/var/lib/kubelet/pods/0b19f8a9-1528-4a10-8935-afb3a4ef2ee7/volumes"
Nov 01 22:28:55.626276 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:28:55.626243    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 22:29:04.817072 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:04.817036    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:29:15.643656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:15.643615    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 22:29:15.650423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:15.650395    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f99686fb5-2wbs5"
Nov 01 22:29:19.549036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:19.549000    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 22:29:19.549837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:19.549806    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy" containerID="cri-o://77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440" gracePeriod=30
Nov 01 22:29:24.888421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.888397    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 22:29:24.902195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.902172    5845 generic.go:356] "Generic (PLEG): container finished" podID="c5e40483-a711-464c-b026-15e3d5b938e6" containerID="77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440" exitCode=0
Nov 01 22:29:24.902350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.902215    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"
Nov 01 22:29:24.902350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.902255    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" event={"ID":"c5e40483-a711-464c-b026-15e3d5b938e6","Type":"ContainerDied","Data":"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"}
Nov 01 22:29:24.902350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.902321    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc" event={"ID":"c5e40483-a711-464c-b026-15e3d5b938e6","Type":"ContainerDied","Data":"1eec66a9edead89e7445a587414d94768cfce044c33a0f8bb75162aabb1b0b86"}
Nov 01 22:29:24.902350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.902346    5845 scope.go:122] "RemoveContainer" containerID="77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"
Nov 01 22:29:24.927920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.927895    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.927988    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928044    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928194 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928179    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928268 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928254    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928360    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928390    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928498    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.928558 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.928544    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\") pod \"c5e40483-a711-464c-b026-15e3d5b938e6\" (UID: \"c5e40483-a711-464c-b026-15e3d5b938e6\") "
Nov 01 22:29:24.929930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.929690    5845 scope.go:122] "RemoveContainer" containerID="77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"
Nov 01 22:29:24.930449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.930415    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:24.931286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.931232    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:24.931771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.931738    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:24.931877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.931822    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:24.932214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.932187    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:24.933187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.933157    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:24.933748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.933720    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:29:24.933748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.933737    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "kube-api-access-wgxmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:24.934052 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.934026    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy" pod "c5e40483-a711-464c-b026-15e3d5b938e6" (UID: "c5e40483-a711-464c-b026-15e3d5b938e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:24.936688 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:24.936658    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440\": container with ID starting with 77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440 not found: ID does not exist" containerID="77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"
Nov 01 22:29:24.936769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:24.936701    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440"} err="failed to get container status \"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440\": rpc error: code = NotFound desc = could not find container \"77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440\": container with ID starting with 77e5e0a6f5ecdbe179df19607019333e6dd089e3a4a523aca823c3ac04bf1440 not found: ID does not exist"
Nov 01 22:29:25.031231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031201    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031229    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031239    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031252    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031261    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgxmc\" (UniqueName: \"kubernetes.io/projected/c5e40483-a711-464c-b026-15e3d5b938e6-kube-api-access-wgxmc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031283    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031292    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e40483-a711-464c-b026-15e3d5b938e6-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031320    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e40483-a711-464c-b026-15e3d5b938e6-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.031440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.031333    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e40483-a711-464c-b026-15e3d5b938e6-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:25.237760 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.237383    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 22:29:25.240937 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:25.240905    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fb9597-openshift-default-749f7c6fc7-l7wcc"]
Nov 01 22:29:26.863950 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:26.863917    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" path="/var/lib/kubelet/pods/c5e40483-a711-464c-b026-15e3d5b938e6/volumes"
Nov 01 22:29:28.163311 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.163261    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:29:28.164375 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.163539    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" podUID="1c27a609-e302-48a1-8988-fb7af5ec4c39" containerName="payload-processing" containerID="cri-o://1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f" gracePeriod=30
Nov 01 22:29:28.189412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.189381    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:29:28.190465 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.190436    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" podUID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" containerName="payload-pre-processing" containerID="cri-o://76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938" gracePeriod=30
Nov 01 22:29:28.681593 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.681513    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:29:28.694666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.694639    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:29:28.869993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.869967    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\") pod \"1c27a609-e302-48a1-8988-fb7af5ec4c39\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") "
Nov 01 22:29:28.870184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870126    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\" (UniqueName: \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\") pod \"1c27a609-e302-48a1-8988-fb7af5ec4c39\" (UID: \"1c27a609-e302-48a1-8988-fb7af5ec4c39\") "
Nov 01 22:29:28.870184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870166    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\" (UniqueName: \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\") pod \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") "
Nov 01 22:29:28.870371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870282    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\") pod \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\" (UID: \"63df327d-8632-4a21-b0e9-9c4127eaa5e6\") "
Nov 01 22:29:28.870473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870443    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume" pod "1c27a609-e302-48a1-8988-fb7af5ec4c39" (UID: "1c27a609-e302-48a1-8988-fb7af5ec4c39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:28.870684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870663    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume" pod "63df327d-8632-4a21-b0e9-9c4127eaa5e6" (UID: "63df327d-8632-4a21-b0e9-9c4127eaa5e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:28.870853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870825    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63df327d-8632-4a21-b0e9-9c4127eaa5e6-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:28.870939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.870853    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c27a609-e302-48a1-8988-fb7af5ec4c39-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:28.872429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.872407    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf" pod "63df327d-8632-4a21-b0e9-9c4127eaa5e6" (UID: "63df327d-8632-4a21-b0e9-9c4127eaa5e6"). InnerVolumeSpecName "kube-api-access-p4tgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:28.872516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.872437    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc" pod "1c27a609-e302-48a1-8988-fb7af5ec4c39" (UID: "1c27a609-e302-48a1-8988-fb7af5ec4c39"). InnerVolumeSpecName "kube-api-access-xwpqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:28.933466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.933365    5845 generic.go:356] "Generic (PLEG): container finished" podID="1c27a609-e302-48a1-8988-fb7af5ec4c39" containerID="1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f" exitCode=0
Nov 01 22:29:28.933466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.933408    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"
Nov 01 22:29:28.933466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.933449    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" event={"ID":"1c27a609-e302-48a1-8988-fb7af5ec4c39","Type":"ContainerDied","Data":"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"}
Nov 01 22:29:28.933782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.933501    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh" event={"ID":"1c27a609-e302-48a1-8988-fb7af5ec4c39","Type":"ContainerDied","Data":"1864bdc3024aae4eba01c897e8dce9b8cfd45f16b03fc2423aa9acbb80b587c1"}
Nov 01 22:29:28.933782 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.933528    5845 scope.go:122] "RemoveContainer" containerID="1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"
Nov 01 22:29:28.935274 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.935242    5845 generic.go:356] "Generic (PLEG): container finished" podID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" containerID="76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938" exitCode=0
Nov 01 22:29:28.935458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.935283    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"
Nov 01 22:29:28.935458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.935351    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" event={"ID":"63df327d-8632-4a21-b0e9-9c4127eaa5e6","Type":"ContainerDied","Data":"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"}
Nov 01 22:29:28.935458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.935390    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr" event={"ID":"63df327d-8632-4a21-b0e9-9c4127eaa5e6","Type":"ContainerDied","Data":"6532ba81961582ac9482d6d329b081d016440aea7fe72a09248c3f7a3f812c1f"}
Nov 01 22:29:28.945609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.945587    5845 scope.go:122] "RemoveContainer" containerID="1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"
Nov 01 22:29:28.945902 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:28.945878    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f\": container with ID starting with 1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f not found: ID does not exist" containerID="1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"
Nov 01 22:29:28.945999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.945914    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f"} err="failed to get container status \"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f\": rpc error: code = NotFound desc = could not find container \"1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f\": container with ID starting with 1c08240429f6ea5ed11e3560b55c2882b275b8c5d4febce6339a870ebfef523f not found: ID does not exist"
Nov 01 22:29:28.945999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.945936    5845 scope.go:122] "RemoveContainer" containerID="76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"
Nov 01 22:29:28.955429 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.955409    5845 scope.go:122] "RemoveContainer" containerID="76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"
Nov 01 22:29:28.955688 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:28.955666    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938\": container with ID starting with 76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938 not found: ID does not exist" containerID="76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"
Nov 01 22:29:28.955749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.955693    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938"} err="failed to get container status \"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938\": rpc error: code = NotFound desc = could not find container \"76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938\": container with ID starting with 76ed65b2d07f105d50c14be401baecd9ef506c5896f474fff3ad44e80e357938 not found: ID does not exist"
Nov 01 22:29:28.965828 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.965803    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:29:28.969615 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.969594    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a68ac-776d8bc74d-z7jxh"]
Nov 01 22:29:28.972077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.972055    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwpqc\" (UniqueName: \"kubernetes.io/projected/1c27a609-e302-48a1-8988-fb7af5ec4c39-kube-api-access-xwpqc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:28.972161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.972078    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tgf\" (UniqueName: \"kubernetes.io/projected/63df327d-8632-4a21-b0e9-9c4127eaa5e6-kube-api-access-p4tgf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:28.980467 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.980447    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:29:28.987125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:28.987106    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a68ac-b78587d55-zjsdr"]
Nov 01 22:29:30.863742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:30.863711    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c27a609-e302-48a1-8988-fb7af5ec4c39" path="/var/lib/kubelet/pods/1c27a609-e302-48a1-8988-fb7af5ec4c39/volumes"
Nov 01 22:29:30.864220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:30.864120    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" path="/var/lib/kubelet/pods/63df327d-8632-4a21-b0e9-9c4127eaa5e6/volumes"
Nov 01 22:29:34.817111 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:34.817028    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:29:36.583981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:36.583950    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:29:36.584495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:36.584178    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" containerID="cri-o://8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953" gracePeriod=30
Nov 01 22:29:37.028547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.028505    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:29:37.029267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029245    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main"
Nov 01 22:29:37.029267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029267    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029319    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029328    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029340    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7c6e92fd-b731-481a-a0ef-bb81369253be" containerName="payload-pre-processing"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029348    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6e92fd-b731-481a-a0ef-bb81369253be" containerName="payload-pre-processing"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029364    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029370    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029380    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029386    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029409    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029415    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029426    5845 cpu_manager.go:414] "containerMap: removing container" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029433    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029447    5845 cpu_manager.go:414] "containerMap: removing container" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy"
Nov 01 22:29:37.029476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029456    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029491    5845 cpu_manager.go:414] "containerMap: removing container" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029498    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029512    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1c27a609-e302-48a1-8988-fb7af5ec4c39" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029519    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c27a609-e302-48a1-8988-fb7af5ec4c39" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029563    5845 cpu_manager.go:414] "containerMap: removing container" podUID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" containerName="payload-pre-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029571    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" containerName="payload-pre-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029587    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029594    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029760    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="468b5cf4-14a0-4954-8c5b-870be68ecf1e" containerName="payload-pre-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029774    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="637a4546-919b-448a-80da-944af0aaf4b3" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029788    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce9762c-ab3f-434e-ad01-db18977833d3" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029799    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c27a609-e302-48a1-8988-fb7af5ec4c39" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029816    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="63df327d-8632-4a21-b0e9-9c4127eaa5e6" containerName="payload-pre-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029825    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99683f0-28b4-48cd-9b26-6a386ba7d1f2" containerName="payload-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029834    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5419e4-dceb-4777-b863-cddeb99eb425" containerName="main"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029846    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdc7f29-1c6d-4c2d-a758-9f58b124c5bf" containerName="main"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029856    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6e92fd-b731-481a-a0ef-bb81369253be" containerName="payload-pre-processing"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029865    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b321c879-897d-4097-b678-9b37339df479" containerName="main"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029874    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e40483-a711-464c-b026-15e3d5b938e6" containerName="istio-proxy"
Nov 01 22:29:37.030184 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.029885    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b19f8a9-1528-4a10-8935-afb3a4ef2ee7" containerName="maas-api"
Nov 01 22:29:37.035736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.035713    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.042524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.042499    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60807f-dockercfg-lf859\""
Nov 01 22:29:37.043332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.043258    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60807f\""
Nov 01 22:29:37.043623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.043596    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:29:37.054389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.054359    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:29:37.059942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.059924    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.086209 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.086178    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:29:37.174323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.170122    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.174323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.170283    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.174323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.170512    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqnq\" (UniqueName: \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.174323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.170588    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksmm\" (UniqueName: \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.277240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.275756    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.277240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.275857    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.277240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.276070    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqnq\" (UniqueName: \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.277240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.276141    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksmm\" (UniqueName: \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.278109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.278083    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.278727 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.278668    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.288749 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.288724    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksmm\" (UniqueName: \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\") pod \"payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.289910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.289886    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqnq\" (UniqueName: \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\") pod \"payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.365144 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.365113    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:37.391506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.391477    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:37.652009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.651984    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:29:37.652615 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:29:37.652585    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfe9ea8_4374_4797_a4b6_fdaca5c675ba.slice/crio-e0ce03d5beba72a5ddfad313505f0bbf24de313582d6c172904790e0bc6f84a6 WatchSource:0}: Error finding container e0ce03d5beba72a5ddfad313505f0bbf24de313582d6c172904790e0bc6f84a6: Status 404 returned error can't find the container with id e0ce03d5beba72a5ddfad313505f0bbf24de313582d6c172904790e0bc6f84a6
Nov 01 22:29:37.686459 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.686431    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:29:37.690686 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:29:37.690661    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664e03b1_bf13_4e9c_b606_ccea7580b2e0.slice/crio-b694b51b887e95272dd8599688edc6d9ab4f9a4c4cb8f450be63a5a8b1f137e9 WatchSource:0}: Error finding container b694b51b887e95272dd8599688edc6d9ab4f9a4c4cb8f450be63a5a8b1f137e9: Status 404 returned error can't find the container with id b694b51b887e95272dd8599688edc6d9ab4f9a4c4cb8f450be63a5a8b1f137e9
Nov 01 22:29:37.799720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.799691    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:29:37.799868 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:37.799741    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:29:38.098795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.098753    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" event={"ID":"664e03b1-bf13-4e9c-b606-ccea7580b2e0","Type":"ContainerStarted","Data":"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"}
Nov 01 22:29:38.098988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.098811    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" event={"ID":"664e03b1-bf13-4e9c-b606-ccea7580b2e0","Type":"ContainerStarted","Data":"b694b51b887e95272dd8599688edc6d9ab4f9a4c4cb8f450be63a5a8b1f137e9"}
Nov 01 22:29:38.098988 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.098847    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:38.100579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.100556    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" event={"ID":"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba","Type":"ContainerStarted","Data":"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"}
Nov 01 22:29:38.100718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.100594    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" event={"ID":"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba","Type":"ContainerStarted","Data":"e0ce03d5beba72a5ddfad313505f0bbf24de313582d6c172904790e0bc6f84a6"}
Nov 01 22:29:38.100718 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.100634    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:38.119211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.119169    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" podStartSLOduration=1.119158474 podStartE2EDuration="1.119158474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:38.11844093 +0000 UTC m=+2681.374620688" watchObservedRunningTime="2026-11-01 22:29:38.119158474 +0000 UTC m=+2681.375338236"
Nov 01 22:29:38.138886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:38.138837    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" podStartSLOduration=1.13881986 podStartE2EDuration="1.13881986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:38.135915628 +0000 UTC m=+2681.392095392" watchObservedRunningTime="2026-11-01 22:29:38.13881986 +0000 UTC m=+2681.394999628"
Nov 01 22:29:39.832383 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:39.832353    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:29:39.856063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:39.856023    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:29:41.876327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.876241    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:29:41.960864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.960833    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.960923    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961060 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.960979    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961060    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961104    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961175    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961242    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961278    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.961365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961340    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:41.962198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961348    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\") pod \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\" (UID: \"006371a8-42ac-402e-ba6c-8bd9e13ed5d8\") "
Nov 01 22:29:41.962198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961465    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:41.962198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961483    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:41.962198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961697    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:41.962198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.961801    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:41.962612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.962205    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:41.962612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.962231    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:41.962612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.962245    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:41.962612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.962364    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:41.962612 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.962382    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:41.963235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.963213    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:29:41.963350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.963280    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:41.963468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.963446    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:29:41.963624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:41.963594    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf" pod "006371a8-42ac-402e-ba6c-8bd9e13ed5d8" (UID: "006371a8-42ac-402e-ba6c-8bd9e13ed5d8"). InnerVolumeSpecName "kube-api-access-k27kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:42.063903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.063809    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:42.063903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.063849    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:42.063903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.063864    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27kf\" (UniqueName: \"kubernetes.io/projected/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-kube-api-access-k27kf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:42.063903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.063878    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006371a8-42ac-402e-ba6c-8bd9e13ed5d8-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:42.172400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.172272    5845 generic.go:356] "Generic (PLEG): container finished" podID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerID="8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953" exitCode=0
Nov 01 22:29:42.172400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.172331    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerDied","Data":"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"}
Nov 01 22:29:42.172400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.172363    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"
Nov 01 22:29:42.172706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.172400    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n" event={"ID":"006371a8-42ac-402e-ba6c-8bd9e13ed5d8","Type":"ContainerDied","Data":"12e6936efb0ae3e1ec736efe92c3a8da503698a44116830b4075a77082df695d"}
Nov 01 22:29:42.172706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.172433    5845 scope.go:122] "RemoveContainer" containerID="8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"
Nov 01 22:29:42.184286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.184263    5845 scope.go:122] "RemoveContainer" containerID="ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"
Nov 01 22:29:42.195233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.195213    5845 scope.go:122] "RemoveContainer" containerID="8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"
Nov 01 22:29:42.195511 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:42.195492    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953\": container with ID starting with 8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953 not found: ID does not exist" containerID="8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"
Nov 01 22:29:42.195595 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.195518    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953"} err="failed to get container status \"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953\": rpc error: code = NotFound desc = could not find container \"8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953\": container with ID starting with 8933df131f26f22adaacaff10d2fcc7d446b82f6d5a128156fd2127117859953 not found: ID does not exist"
Nov 01 22:29:42.195595 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.195532    5845 scope.go:122] "RemoveContainer" containerID="ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"
Nov 01 22:29:42.195805 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:29:42.195789    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25\": container with ID starting with ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25 not found: ID does not exist" containerID="ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"
Nov 01 22:29:42.195852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.195810    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25"} err="failed to get container status \"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25\": rpc error: code = NotFound desc = could not find container \"ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25\": container with ID starting with ff1f5ceb7c7823caebe685fde348620554649ef1066be38aa6b27206d3679d25 not found: ID does not exist"
Nov 01 22:29:42.197258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.197238    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:29:42.204709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.204688    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a68ac-openshift-default-56bd457b8f-tsq9n"]
Nov 01 22:29:42.867062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:42.867020    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" path="/var/lib/kubelet/pods/006371a8-42ac-402e-ba6c-8bd9e13ed5d8/volumes"
Nov 01 22:29:44.832659 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:44.832615    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" containerID="cri-o://01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007" gracePeriod=25
Nov 01 22:29:44.856942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:44.856889    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" containerID="cri-o://c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c" gracePeriod=25
Nov 01 22:29:47.611439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:47.611402    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:29:48.620104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:48.620063    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:29:49.048560 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.048525    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:29:49.049630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049605    5845 cpu_manager.go:414] "containerMap: removing container" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.049630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049630    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.049793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049670    5845 cpu_manager.go:414] "containerMap: removing container" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.049793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049679    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.049899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049884    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.049936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.049910    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="006371a8-42ac-402e-ba6c-8bd9e13ed5d8" containerName="istio-proxy"
Nov 01 22:29:49.057008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.056983    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.063997 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.063977    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-60807f-serving-cert\""
Nov 01 22:29:49.068349 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.068320    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:29:49.111199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.111175    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:29:49.111370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.111241    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:29:49.194501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.194453    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.194895 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.194865    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hm5\" (UniqueName: \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.195012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.194937    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.195221 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.195195    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.296591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.296557    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.296764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.296708    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.296823 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.296809    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.296923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.296907    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hm5\" (UniqueName: \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.297428 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.297404    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.299085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.299036    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.299165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.299094    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.308976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.308948    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hm5\" (UniqueName: \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\") pod \"maas-api-e2e-ipp-b-60807f-667857dd76-86nbp\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.379447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.379418    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:49.694595 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.694535    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:29:49.696042 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:29:49.696002    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfdb033_56bf_41e6_93df_e8a66e3de202.slice/crio-311218d2239ce5320f04f208f58fcc74e81ae5a8d38a8371bb6b7110bafa4f7e WatchSource:0}: Error finding container 311218d2239ce5320f04f208f58fcc74e81ae5a8d38a8371bb6b7110bafa4f7e: Status 404 returned error can't find the container with id 311218d2239ce5320f04f208f58fcc74e81ae5a8d38a8371bb6b7110bafa4f7e
Nov 01 22:29:49.697962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:49.697941    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:29:50.238720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:50.238689    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" event={"ID":"1dfdb033-56bf-41e6-93df-e8a66e3de202","Type":"ContainerStarted","Data":"311218d2239ce5320f04f208f58fcc74e81ae5a8d38a8371bb6b7110bafa4f7e"}
Nov 01 22:29:51.248907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:51.248875    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" event={"ID":"1dfdb033-56bf-41e6-93df-e8a66e3de202","Type":"ContainerStarted","Data":"2bd7c323d3a631c5944bda084fba0b7d7989d5b10a5cf3d46be075c8305417f7"}
Nov 01 22:29:51.249343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:51.248963    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:51.274098 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:51.274050    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" podStartSLOduration=1.7863540009999999 podStartE2EDuration="2.274035278s" totalImagesPullingTime="487.681277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:51.270263888 +0000 UTC m=+2694.526443663" watchObservedRunningTime="2026-11-01 22:29:51.274035278 +0000 UTC m=+2694.530215040"
Nov 01 22:29:52.026688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.026089    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:29:52.026688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.026361    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api" containerID="cri-o://eb329e2225445ae6c8d436eed8d15f502da9a2be9b952a837f11a83935dcc4bd" gracePeriod=30
Nov 01 22:29:52.346380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.330003    5845 generic.go:356] "Generic (PLEG): container finished" podID="3c22bc83-8cab-44be-bead-1115064df300" containerID="eb329e2225445ae6c8d436eed8d15f502da9a2be9b952a837f11a83935dcc4bd" exitCode=0
Nov 01 22:29:52.346380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.331183    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" event={"ID":"3c22bc83-8cab-44be-bead-1115064df300","Type":"ContainerDied","Data":"eb329e2225445ae6c8d436eed8d15f502da9a2be9b952a837f11a83935dcc4bd"}
Nov 01 22:29:52.548321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.542938    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:29:52.631913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.630845    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\" (UniqueName: \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\") pod \"3c22bc83-8cab-44be-bead-1115064df300\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") "
Nov 01 22:29:52.631913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.630945    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") pod \"3c22bc83-8cab-44be-bead-1115064df300\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") "
Nov 01 22:29:52.631913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.631164    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\") pod \"3c22bc83-8cab-44be-bead-1115064df300\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") "
Nov 01 22:29:52.631913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.631225    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\") pod \"3c22bc83-8cab-44be-bead-1115064df300\" (UID: \"3c22bc83-8cab-44be-bead-1115064df300\") "
Nov 01 22:29:52.633717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.633594    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca" pod "3c22bc83-8cab-44be-bead-1115064df300" (UID: "3c22bc83-8cab-44be-bead-1115064df300"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:29:52.636150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.636126    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb" pod "3c22bc83-8cab-44be-bead-1115064df300" (UID: "3c22bc83-8cab-44be-bead-1115064df300"). InnerVolumeSpecName "kube-api-access-8zntb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:52.638744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.638704    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs" pod "3c22bc83-8cab-44be-bead-1115064df300" (UID: "3c22bc83-8cab-44be-bead-1115064df300"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:29:52.638887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.638722    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls" pod "3c22bc83-8cab-44be-bead-1115064df300" (UID: "3c22bc83-8cab-44be-bead-1115064df300"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:29:52.734000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.733967    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:52.734000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.733996    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c22bc83-8cab-44be-bead-1115064df300-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:52.734000 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.734006    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zntb\" (UniqueName: \"kubernetes.io/projected/3c22bc83-8cab-44be-bead-1115064df300-kube-api-access-8zntb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:52.734244 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:52.734017    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c22bc83-8cab-44be-bead-1115064df300-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:53.339695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:53.339666    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"
Nov 01 22:29:53.339875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:53.339693    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t" event={"ID":"3c22bc83-8cab-44be-bead-1115064df300","Type":"ContainerDied","Data":"b7380f1a5100858aaedb8cc07d91d293385dd4d2a6f716563c2f288ccbdad974"}
Nov 01 22:29:53.339875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:53.339777    5845 scope.go:122] "RemoveContainer" containerID="eb329e2225445ae6c8d436eed8d15f502da9a2be9b952a837f11a83935dcc4bd"
Nov 01 22:29:53.358865 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:53.358840    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:29:53.363764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:53.363741    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e3890e-9d65c8669-df68t"]
Nov 01 22:29:54.862841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:54.862801    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c22bc83-8cab-44be-bead-1115064df300" path="/var/lib/kubelet/pods/3c22bc83-8cab-44be-bead-1115064df300/volumes"
Nov 01 22:29:57.337936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:57.337910    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:29:57.612224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:57.612143    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:29:58.619969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:58.619926    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:29:59.262713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.262684    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:29:59.300598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.300566    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:29:59.313911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.313887    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:29:59.318668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.318646    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:29:59.527970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.527875    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:29:59.528196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.528156    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy" containerID="cri-o://0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724" gracePeriod=30
Nov 01 22:29:59.533289 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.533246    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:29:59.533534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:29:59.533507    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy" containerID="cri-o://6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136" gracePeriod=30
Nov 01 22:30:00.201999 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.201964    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:00.202703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.202682    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api"
Nov 01 22:30:00.202703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.202704    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api"
Nov 01 22:30:00.202982 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.202966    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c22bc83-8cab-44be-bead-1115064df300" containerName="maas-api"
Nov 01 22:30:00.207411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.207389    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.264094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.263668    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mvwcj\""
Nov 01 22:30:00.264094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.263715    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:00.267431 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.267397    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjdm\" (UniqueName: \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.267786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.267758    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.269340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.269318    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.270013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.269993    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:00.276367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.275729    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:00.298912 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.298883    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:00.315220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.314446    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:00.315220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.314559    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.369516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369485    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjdm\" (UniqueName: \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.369666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369552    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.369666 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369625    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s8k\" (UniqueName: \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.369783 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369760    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.369819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369809    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.369862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.369845    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd56f\" (UniqueName: \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.370330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.370314    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.381758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.381729    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjdm\" (UniqueName: \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\") pod \"maas-api-key-cleanup-29892870-tn7rj\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.471136 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471103    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.471366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471158    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd56f\" (UniqueName: \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.471366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471337    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.472126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471417    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s8k\" (UniqueName: \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.472126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471819    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.472126 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.471956    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.483766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.483736    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd56f\" (UniqueName: \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\") pod \"maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.483911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.483888    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s8k\" (UniqueName: \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\") pod \"maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.544983 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.544957    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:00.615571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.615542    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:00.634942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.632018    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:00.837521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.837493    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:00.843393 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:30:00.840894    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b88531_bad5_465a_8275_01770308a874.slice/crio-984f24c06500a1d8d4a2f950734c5fe10c10ebbe6409b4ce5846a9c0e8364f13 WatchSource:0}: Error finding container 984f24c06500a1d8d4a2f950734c5fe10c10ebbe6409b4ce5846a9c0e8364f13: Status 404 returned error can't find the container with id 984f24c06500a1d8d4a2f950734c5fe10c10ebbe6409b4ce5846a9c0e8364f13
Nov 01 22:30:00.922461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.922425    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:00.923191 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:30:00.923160    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05e232c_d130_4b79_a95a_009bc1492770.slice/crio-6da839c9d4c285989a31ce599359eb21c0fd0c89d1eba617cc9c26acc97d6f62 WatchSource:0}: Error finding container 6da839c9d4c285989a31ce599359eb21c0fd0c89d1eba617cc9c26acc97d6f62: Status 404 returned error can't find the container with id 6da839c9d4c285989a31ce599359eb21c0fd0c89d1eba617cc9c26acc97d6f62
Nov 01 22:30:00.990744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:00.990716    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:00.991190 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:30:00.991167    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614512c6_d381_48ee_845c_fc429ba794b9.slice/crio-15eb555a0d8c292f3e049dbe784ab0f1c5a45cd938b9b97dfc2ff7539c65fb4f WatchSource:0}: Error finding container 15eb555a0d8c292f3e049dbe784ab0f1c5a45cd938b9b97dfc2ff7539c65fb4f: Status 404 returned error can't find the container with id 15eb555a0d8c292f3e049dbe784ab0f1c5a45cd938b9b97dfc2ff7539c65fb4f
Nov 01 22:30:01.421366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:01.421328    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerStarted","Data":"15eb555a0d8c292f3e049dbe784ab0f1c5a45cd938b9b97dfc2ff7539c65fb4f"}
Nov 01 22:30:01.423347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:01.423318    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerStarted","Data":"ca22128b5e283cad4d1cc6c0364e82b5fd0fb92c1245eac31a8b0f03d66d1ad7"}
Nov 01 22:30:01.423519 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:01.423372    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerStarted","Data":"984f24c06500a1d8d4a2f950734c5fe10c10ebbe6409b4ce5846a9c0e8364f13"}
Nov 01 22:30:01.424726 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:01.424706    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerStarted","Data":"6da839c9d4c285989a31ce599359eb21c0fd0c89d1eba617cc9c26acc97d6f62"}
Nov 01 22:30:01.443862 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:01.443807    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" podStartSLOduration=1.443791533 podStartE2EDuration="1.443791533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:01.44262892 +0000 UTC m=+2704.698808684" watchObservedRunningTime="2026-11-01 22:30:01.443791533 +0000 UTC m=+2704.699971302"
Nov 01 22:30:02.435071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.434973    5845 generic.go:356] "Generic (PLEG): container finished" podID="614512c6-d381-48ee-845c-fc429ba794b9" containerID="f0a33da29a9af45fb084b821b9385c5369d94d2eb0798304c8a1da2fc64e7b99" exitCode=60
Nov 01 22:30:02.435641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.435063    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerDied","Data":"f0a33da29a9af45fb084b821b9385c5369d94d2eb0798304c8a1da2fc64e7b99"}
Nov 01 22:30:02.435641 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.435485    5845 scope.go:122] "RemoveContainer" containerID="f0a33da29a9af45fb084b821b9385c5369d94d2eb0798304c8a1da2fc64e7b99"
Nov 01 22:30:02.438547 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.438526    5845 generic.go:356] "Generic (PLEG): container finished" podID="97b88531-bad5-465a-8275-01770308a874" containerID="ca22128b5e283cad4d1cc6c0364e82b5fd0fb92c1245eac31a8b0f03d66d1ad7" exitCode=60
Nov 01 22:30:02.438687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.438669    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerDied","Data":"ca22128b5e283cad4d1cc6c0364e82b5fd0fb92c1245eac31a8b0f03d66d1ad7"}
Nov 01 22:30:02.439053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.439036    5845 scope.go:122] "RemoveContainer" containerID="ca22128b5e283cad4d1cc6c0364e82b5fd0fb92c1245eac31a8b0f03d66d1ad7"
Nov 01 22:30:02.480382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.476984    5845 generic.go:356] "Generic (PLEG): container finished" podID="a05e232c-d130-4b79-a95a-009bc1492770" containerID="fe4f289016e8f4e1676ccb2068f65176f5f0555c0839b1aea25408794fd0caf5" exitCode=60
Nov 01 22:30:02.480382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.477057    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerDied","Data":"fe4f289016e8f4e1676ccb2068f65176f5f0555c0839b1aea25408794fd0caf5"}
Nov 01 22:30:02.480382 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:02.477552    5845 scope.go:122] "RemoveContainer" containerID="fe4f289016e8f4e1676ccb2068f65176f5f0555c0839b1aea25408794fd0caf5"
Nov 01 22:30:03.487483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.487451    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerStarted","Data":"124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"}
Nov 01 22:30:03.489864 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.489547    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerStarted","Data":"c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"}
Nov 01 22:30:03.491831 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.491808    5845 generic.go:356] "Generic (PLEG): container finished" podID="a05e232c-d130-4b79-a95a-009bc1492770" containerID="5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971" exitCode=60
Nov 01 22:30:03.491949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.491842    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerDied","Data":"5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971"}
Nov 01 22:30:03.491949 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.491893    5845 scope.go:122] "RemoveContainer" containerID="fe4f289016e8f4e1676ccb2068f65176f5f0555c0839b1aea25408794fd0caf5"
Nov 01 22:30:03.492153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.492136    5845 scope.go:122] "RemoveContainer" containerID="5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971"
Nov 01 22:30:03.492520 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:03.492493    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn_odh-ai-gateway-infra(a05e232c-d130-4b79-a95a-009bc1492770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" podUID="a05e232c-d130-4b79-a95a-009bc1492770"
Nov 01 22:30:03.507400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:03.507361    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" podStartSLOduration=3.5073485939999998 podStartE2EDuration="3.507348594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:03.505061956 +0000 UTC m=+2706.761241719" watchObservedRunningTime="2026-11-01 22:30:03.507348594 +0000 UTC m=+2706.763528350"
Nov 01 22:30:04.501732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.501699    5845 generic.go:356] "Generic (PLEG): container finished" podID="97b88531-bad5-465a-8275-01770308a874" containerID="c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d" exitCode=60
Nov 01 22:30:04.502217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.501788    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerDied","Data":"c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"}
Nov 01 22:30:04.502217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.501935    5845 scope.go:122] "RemoveContainer" containerID="ca22128b5e283cad4d1cc6c0364e82b5fd0fb92c1245eac31a8b0f03d66d1ad7"
Nov 01 22:30:04.502217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.502040    5845 scope.go:122] "RemoveContainer" containerID="c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"
Nov 01 22:30:04.502519 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:04.502479    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-tn7rj_odh-ai-gateway-infra(97b88531-bad5-465a-8275-01770308a874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" podUID="97b88531-bad5-465a-8275-01770308a874"
Nov 01 22:30:04.503888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.503871    5845 scope.go:122] "RemoveContainer" containerID="5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971"
Nov 01 22:30:04.504188 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:04.504163    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn_odh-ai-gateway-infra(a05e232c-d130-4b79-a95a-009bc1492770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" podUID="a05e232c-d130-4b79-a95a-009bc1492770"
Nov 01 22:30:04.505811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.505787    5845 generic.go:356] "Generic (PLEG): container finished" podID="614512c6-d381-48ee-845c-fc429ba794b9" containerID="124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b" exitCode=60
Nov 01 22:30:04.505888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.505828    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerDied","Data":"124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"}
Nov 01 22:30:04.506102 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.506081    5845 scope.go:122] "RemoveContainer" containerID="124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"
Nov 01 22:30:04.506362 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:04.506343    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5_odh-ai-gateway-infra(614512c6-d381-48ee-845c-fc429ba794b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" podUID="614512c6-d381-48ee-845c-fc429ba794b9"
Nov 01 22:30:04.515152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.515131    5845 scope.go:122] "RemoveContainer" containerID="f0a33da29a9af45fb084b821b9385c5369d94d2eb0798304c8a1da2fc64e7b99"
Nov 01 22:30:04.834673 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:04.834639    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:30:04.848576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.848549    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:30:04.956044 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956006    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.956356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956333    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.956567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956551    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.956736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956718    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.956850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956836    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.956966 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.956952    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.957088 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.957075    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.957455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.957431    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.957947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.957925    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\") pod \"67c66cc6-05aa-449e-aa44-618406f83882\" (UID: \"67c66cc6-05aa-449e-aa44-618406f83882\") "
Nov 01 22:30:04.970114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.966016    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:04.970114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.967774    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:04.970114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.969481    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:04.970114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.969834    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:04.970614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.970561    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:04.981204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.981168    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:04.981495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.981462    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:04.981732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:04.981700    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "kube-api-access-bp94l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:05.016897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.016861    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo" pod "67c66cc6-05aa-449e-aa44-618406f83882" (UID: "67c66cc6-05aa-449e-aa44-618406f83882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:30:05.054387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.046902    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061458    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp94l\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-kube-api-access-bp94l\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061492    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061507    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061518    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061530    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c66cc6-05aa-449e-aa44-618406f83882-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061543    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c66cc6-05aa-449e-aa44-618406f83882-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061556    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061570    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c66cc6-05aa-449e-aa44-618406f83882-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.062224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.061583    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c66cc6-05aa-449e-aa44-618406f83882-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.162939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.162894    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.162965    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163093    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163206    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163279    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163326    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163380    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163414    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.163592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.163533    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\") pod \"b2596176-d07d-49e5-b87f-b285e49045a6\" (UID: \"b2596176-d07d-49e5-b87f-b285e49045a6\") "
Nov 01 22:30:05.164325 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.164133    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:05.164434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.164370    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:05.164434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.164384    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:05.164883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.164654    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:05.164883 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.164850    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:05.167139 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.167110    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:05.170648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.170618    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:05.170742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.170671    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "kube-api-access-gmjxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:05.172676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.172649    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo" pod "b2596176-d07d-49e5-b87f-b285e49045a6" (UID: "b2596176-d07d-49e5-b87f-b285e49045a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:30:05.265434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265396    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265435    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265451    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265465    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265480    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265494    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjxf\" (UniqueName: \"kubernetes.io/projected/b2596176-d07d-49e5-b87f-b285e49045a6-kube-api-access-gmjxf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265508    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2596176-d07d-49e5-b87f-b285e49045a6-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265524    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2596176-d07d-49e5-b87f-b285e49045a6-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.265696 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.265536    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2596176-d07d-49e5-b87f-b285e49045a6-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.295420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.295398    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:30:05.300813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.300790    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:30:05.367003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.366970    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\" (UniqueName: \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\") pod \"19d7cba2-325a-48db-99da-27c4d5b27e90\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") "
Nov 01 22:30:05.367161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367039    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\" (UniqueName: \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\") pod \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") "
Nov 01 22:30:05.367161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367071    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\") pod \"19d7cba2-325a-48db-99da-27c4d5b27e90\" (UID: \"19d7cba2-325a-48db-99da-27c4d5b27e90\") "
Nov 01 22:30:05.367161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367102    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\") pod \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\" (UID: \"b5a76f1f-91d6-4c58-b228-bb7d44a660a3\") "
Nov 01 22:30:05.367543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367504    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume" pod "19d7cba2-325a-48db-99da-27c4d5b27e90" (UID: "19d7cba2-325a-48db-99da-27c4d5b27e90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:05.367805 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367785    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19d7cba2-325a-48db-99da-27c4d5b27e90-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.367965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.367942    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume" pod "b5a76f1f-91d6-4c58-b228-bb7d44a660a3" (UID: "b5a76f1f-91d6-4c58-b228-bb7d44a660a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:05.369371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.369352    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc" pod "19d7cba2-325a-48db-99da-27c4d5b27e90" (UID: "19d7cba2-325a-48db-99da-27c4d5b27e90"). InnerVolumeSpecName "kube-api-access-tstmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:05.369675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.369659    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw" pod "b5a76f1f-91d6-4c58-b228-bb7d44a660a3" (UID: "b5a76f1f-91d6-4c58-b228-bb7d44a660a3"). InnerVolumeSpecName "kube-api-access-7dnhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:05.468961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.468921    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tstmc\" (UniqueName: \"kubernetes.io/projected/19d7cba2-325a-48db-99da-27c4d5b27e90-kube-api-access-tstmc\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.468961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.468955    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnhw\" (UniqueName: \"kubernetes.io/projected/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-kube-api-access-7dnhw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.469177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.468969    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a76f1f-91d6-4c58-b228-bb7d44a660a3-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:05.514877 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.514843    5845 generic.go:356] "Generic (PLEG): container finished" podID="67c66cc6-05aa-449e-aa44-618406f83882" containerID="0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724" exitCode=0
Nov 01 22:30:05.515321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.514893    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"
Nov 01 22:30:05.515321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.514929    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerDied","Data":"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"}
Nov 01 22:30:05.515321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.514982    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l" event={"ID":"67c66cc6-05aa-449e-aa44-618406f83882","Type":"ContainerDied","Data":"85e870bca09b2d48bcf4f36068bb7f8da8884f0495e589bb25b63f8758f71748"}
Nov 01 22:30:05.515321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.515003    5845 scope.go:122] "RemoveContainer" containerID="0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"
Nov 01 22:30:05.516663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.516639    5845 generic.go:356] "Generic (PLEG): container finished" podID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerID="01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007" exitCode=0
Nov 01 22:30:05.516771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.516683    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"
Nov 01 22:30:05.516771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.516730    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" event={"ID":"19d7cba2-325a-48db-99da-27c4d5b27e90","Type":"ContainerDied","Data":"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"}
Nov 01 22:30:05.516881 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.516785    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm" event={"ID":"19d7cba2-325a-48db-99da-27c4d5b27e90","Type":"ContainerDied","Data":"fea66a63fc1cb1288c393ad23213c8892ad95664f2156afc5c77a16ab00090f7"}
Nov 01 22:30:05.518330 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.518271    5845 generic.go:356] "Generic (PLEG): container finished" podID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerID="c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c" exitCode=0
Nov 01 22:30:05.518440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.518332    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"
Nov 01 22:30:05.518440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.518348    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" event={"ID":"b5a76f1f-91d6-4c58-b228-bb7d44a660a3","Type":"ContainerDied","Data":"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"}
Nov 01 22:30:05.518440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.518408    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j" event={"ID":"b5a76f1f-91d6-4c58-b228-bb7d44a660a3","Type":"ContainerDied","Data":"88d78313d1a74defa596d9f07eb490e4dc713a710752b9601cce21bb5b4769a1"}
Nov 01 22:30:05.520900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.520806    5845 scope.go:122] "RemoveContainer" containerID="124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"
Nov 01 22:30:05.521105 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.521077    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5_odh-ai-gateway-infra(614512c6-d381-48ee-845c-fc429ba794b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" podUID="614512c6-d381-48ee-845c-fc429ba794b9"
Nov 01 22:30:05.524269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.524138    5845 generic.go:356] "Generic (PLEG): container finished" podID="b2596176-d07d-49e5-b87f-b285e49045a6" containerID="6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136" exitCode=0
Nov 01 22:30:05.524269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.524163    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" event={"ID":"b2596176-d07d-49e5-b87f-b285e49045a6","Type":"ContainerDied","Data":"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"}
Nov 01 22:30:05.524269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.524187    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"
Nov 01 22:30:05.524269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.524214    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g" event={"ID":"b2596176-d07d-49e5-b87f-b285e49045a6","Type":"ContainerDied","Data":"76aaf90774b68b99a5ce75d2dbef7dcabeb21a4c719b5f8d90deaf32f6720030"}
Nov 01 22:30:05.527168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.527148    5845 scope.go:122] "RemoveContainer" containerID="c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"
Nov 01 22:30:05.527534 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.527476    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-tn7rj_odh-ai-gateway-infra(97b88531-bad5-465a-8275-01770308a874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" podUID="97b88531-bad5-465a-8275-01770308a874"
Nov 01 22:30:05.530152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.529075    5845 scope.go:122] "RemoveContainer" containerID="245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"
Nov 01 22:30:05.542992 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.542977    5845 scope.go:122] "RemoveContainer" containerID="0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"
Nov 01 22:30:05.543249 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.543229    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724\": container with ID starting with 0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724 not found: ID does not exist" containerID="0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"
Nov 01 22:30:05.543360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.543256    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724"} err="failed to get container status \"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724\": rpc error: code = NotFound desc = could not find container \"0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724\": container with ID starting with 0ad854995fe3e877174fd65b8033a03e670202cfb0ccc8f13b81f96dcc896724 not found: ID does not exist"
Nov 01 22:30:05.543360 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.543271    5845 scope.go:122] "RemoveContainer" containerID="245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"
Nov 01 22:30:05.543518 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.543501    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a\": container with ID starting with 245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a not found: ID does not exist" containerID="245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"
Nov 01 22:30:05.543566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.543523    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a"} err="failed to get container status \"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a\": rpc error: code = NotFound desc = could not find container \"245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a\": container with ID starting with 245012a5190d8c30f79e187c7de5d4be2b1b0fd7146db59f03013cded4bcee8a not found: ID does not exist"
Nov 01 22:30:05.543566 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.543536    5845 scope.go:122] "RemoveContainer" containerID="01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"
Nov 01 22:30:05.556993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.556978    5845 scope.go:122] "RemoveContainer" containerID="01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"
Nov 01 22:30:05.557290 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.557263    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007\": container with ID starting with 01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007 not found: ID does not exist" containerID="01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"
Nov 01 22:30:05.557385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.557310    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007"} err="failed to get container status \"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007\": rpc error: code = NotFound desc = could not find container \"01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007\": container with ID starting with 01d4b69b3fb083c513ced9ce56ebb3339ca7be01d336dd79f1fee23b58bde007 not found: ID does not exist"
Nov 01 22:30:05.557385 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.557333    5845 scope.go:122] "RemoveContainer" containerID="c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"
Nov 01 22:30:05.575241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.575217    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:30:05.580756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.580731    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-e3890e-8689f59474-s8blm"]
Nov 01 22:30:05.590112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.590078    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:30:05.613576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.613542    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-e3890e-7898d88d67-jdf6j"]
Nov 01 22:30:05.614397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.614373    5845 scope.go:122] "RemoveContainer" containerID="c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"
Nov 01 22:30:05.614825 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.614803    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c\": container with ID starting with c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c not found: ID does not exist" containerID="c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"
Nov 01 22:30:05.614920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.614833    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c"} err="failed to get container status \"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c\": rpc error: code = NotFound desc = could not find container \"c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c\": container with ID starting with c39c39fb061fb1036542b4c7670e31260df4131cbbad32640d53c557c5b51a2c not found: ID does not exist"
Nov 01 22:30:05.614920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.614856    5845 scope.go:122] "RemoveContainer" containerID="6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"
Nov 01 22:30:05.625815 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.625791    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:30:05.629040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.629018    5845 scope.go:122] "RemoveContainer" containerID="6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"
Nov 01 22:30:05.629378 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:05.629350    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136\": container with ID starting with 6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136 not found: ID does not exist" containerID="6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"
Nov 01 22:30:05.629479 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.629389    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136"} err="failed to get container status \"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136\": rpc error: code = NotFound desc = could not find container \"6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136\": container with ID starting with 6c84972ec4c964160b32b963ee4e2308166eb078ade2045fa91659795363b136 not found: ID does not exist"
Nov 01 22:30:05.631439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.631417    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-9888g"]
Nov 01 22:30:05.644649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.644617    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:30:05.655810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.655787    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e3890e-openshift-default-6975b574b5-8gm5l"]
Nov 01 22:30:05.776201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.776170    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:30:05.788324 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:05.788281    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:30:06.862732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:06.862690    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" path="/var/lib/kubelet/pods/19d7cba2-325a-48db-99da-27c4d5b27e90/volumes"
Nov 01 22:30:06.863381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:06.863358    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c66cc6-05aa-449e-aa44-618406f83882" path="/var/lib/kubelet/pods/67c66cc6-05aa-449e-aa44-618406f83882/volumes"
Nov 01 22:30:06.864030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:06.864008    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" path="/var/lib/kubelet/pods/b2596176-d07d-49e5-b87f-b285e49045a6/volumes"
Nov 01 22:30:06.864779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:06.864757    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" path="/var/lib/kubelet/pods/b5a76f1f-91d6-4c58-b228-bb7d44a660a3/volumes"
Nov 01 22:30:10.109495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:10.109417    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:30:10.776863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:10.776821    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" podUID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" containerName="payload-processing" containerID="cri-o://c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3" gracePeriod=25
Nov 01 22:30:10.789496 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:10.789459    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" podUID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" containerName="payload-pre-processing" containerID="cri-o://9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599" gracePeriod=25
Nov 01 22:30:11.108958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.108930    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:30:11.131600 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.131565    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:30:11.271367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271336    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\" (UniqueName: \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\") pod \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") "
Nov 01 22:30:11.271539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271434    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\" (UniqueName: \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\") pod \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") "
Nov 01 22:30:11.271539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271501    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\") pod \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\" (UID: \"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba\") "
Nov 01 22:30:11.271627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271603    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\") pod \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\" (UID: \"664e03b1-bf13-4e9c-b606-ccea7580b2e0\") "
Nov 01 22:30:11.271919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271892    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume" pod "8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" (UID: "8cfe9ea8-4374-4797-a4b6-fdaca5c675ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:11.272026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.271951    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume" pod "664e03b1-bf13-4e9c-b606-ccea7580b2e0" (UID: "664e03b1-bf13-4e9c-b606-ccea7580b2e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:11.272410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.272388    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/664e03b1-bf13-4e9c-b606-ccea7580b2e0-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:11.272489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.272414    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:11.273637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.273617    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq" pod "8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" (UID: "8cfe9ea8-4374-4797-a4b6-fdaca5c675ba"). InnerVolumeSpecName "kube-api-access-cvqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:11.273698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.273653    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm" pod "664e03b1-bf13-4e9c-b606-ccea7580b2e0" (UID: "664e03b1-bf13-4e9c-b606-ccea7580b2e0"). InnerVolumeSpecName "kube-api-access-6ksmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:11.383103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.382863    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvqnq\" (UniqueName: \"kubernetes.io/projected/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba-kube-api-access-cvqnq\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:11.383103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.382898    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ksmm\" (UniqueName: \"kubernetes.io/projected/664e03b1-bf13-4e9c-b606-ccea7580b2e0-kube-api-access-6ksmm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:11.583955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.583910    5845 generic.go:356] "Generic (PLEG): container finished" podID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" containerID="c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3" exitCode=0
Nov 01 22:30:11.584408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.584374    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" event={"ID":"664e03b1-bf13-4e9c-b606-ccea7580b2e0","Type":"ContainerDied","Data":"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"}
Nov 01 22:30:11.584592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.584422    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"
Nov 01 22:30:11.584592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.584485    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4" event={"ID":"664e03b1-bf13-4e9c-b606-ccea7580b2e0","Type":"ContainerDied","Data":"b694b51b887e95272dd8599688edc6d9ab4f9a4c4cb8f450be63a5a8b1f137e9"}
Nov 01 22:30:11.584592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.584545    5845 scope.go:122] "RemoveContainer" containerID="c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"
Nov 01 22:30:11.646195 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.646158    5845 generic.go:356] "Generic (PLEG): container finished" podID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" containerID="9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599" exitCode=0
Nov 01 22:30:11.646373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.646211    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" event={"ID":"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba","Type":"ContainerDied","Data":"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"}
Nov 01 22:30:11.646373 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.646275    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf" event={"ID":"8cfe9ea8-4374-4797-a4b6-fdaca5c675ba","Type":"ContainerDied","Data":"e0ce03d5beba72a5ddfad313505f0bbf24de313582d6c172904790e0bc6f84a6"}
Nov 01 22:30:11.646511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.646415    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"
Nov 01 22:30:11.693717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.686461    5845 scope.go:122] "RemoveContainer" containerID="c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"
Nov 01 22:30:11.693717 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:11.686916    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3\": container with ID starting with c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3 not found: ID does not exist" containerID="c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"
Nov 01 22:30:11.693717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.686958    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3"} err="failed to get container status \"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3\": rpc error: code = NotFound desc = could not find container \"c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3\": container with ID starting with c629676011ff6f7d21dcb5d709c6df9a15c9c336bae02759b358bf2c3c36dcb3 not found: ID does not exist"
Nov 01 22:30:11.693717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.686982    5845 scope.go:122] "RemoveContainer" containerID="9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"
Nov 01 22:30:11.711619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.711584    5845 scope.go:122] "RemoveContainer" containerID="9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"
Nov 01 22:30:11.712989 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:11.712947    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599\": container with ID starting with 9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599 not found: ID does not exist" containerID="9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"
Nov 01 22:30:11.713199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.713166    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599"} err="failed to get container status \"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599\": rpc error: code = NotFound desc = could not find container \"9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599\": container with ID starting with 9988767023d7a3aa497c2e156d8c7e9af4fb8d5d902f6366eef852be49084599 not found: ID does not exist"
Nov 01 22:30:11.714233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.714213    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:30:11.720707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.720656    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-cfdc57d6d-ns5d4"]
Nov 01 22:30:11.724040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.724018    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:30:11.727328 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:11.727289    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60807f-7d8857bcf9-r87wf"]
Nov 01 22:30:12.897580 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:12.897544    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" path="/var/lib/kubelet/pods/664e03b1-bf13-4e9c-b606-ccea7580b2e0/volumes"
Nov 01 22:30:12.898846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:12.898818    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" path="/var/lib/kubelet/pods/8cfe9ea8-4374-4797-a4b6-fdaca5c675ba/volumes"
Nov 01 22:30:15.110097 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:15.110048    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing" containerID="cri-o://69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b" gracePeriod=25
Nov 01 22:30:15.853917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:15.853889    5845 scope.go:122] "RemoveContainer" containerID="c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"
Nov 01 22:30:15.854132 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:15.853995    5845 scope.go:122] "RemoveContainer" containerID="124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"
Nov 01 22:30:16.515420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:16.515390    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:30:16.686725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:16.686693    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerStarted","Data":"dcb6e48a362410304127c909f53648ab700669726e570552cc0c42cca78ac6e8"}
Nov 01 22:30:16.688793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:16.688769    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerStarted","Data":"72d668ddd74f366fafe61d057c42cd41da6c58a021e9a9a3767056b0ebd96685"}
Nov 01 22:30:17.721404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.720222    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:17.725319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.723475    5845 generic.go:356] "Generic (PLEG): container finished" podID="614512c6-d381-48ee-845c-fc429ba794b9" containerID="dcb6e48a362410304127c909f53648ab700669726e570552cc0c42cca78ac6e8" exitCode=60
Nov 01 22:30:17.725319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.723664    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerDied","Data":"dcb6e48a362410304127c909f53648ab700669726e570552cc0c42cca78ac6e8"}
Nov 01 22:30:17.725319 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.723729    5845 scope.go:122] "RemoveContainer" containerID="124be90be929ddf69f6b0779095baa25c3146212fa19d44b420baa4269f2ed9b"
Nov 01 22:30:17.731471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.730871    5845 generic.go:356] "Generic (PLEG): container finished" podID="97b88531-bad5-465a-8275-01770308a874" containerID="72d668ddd74f366fafe61d057c42cd41da6c58a021e9a9a3767056b0ebd96685" exitCode=60
Nov 01 22:30:17.731471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.731041    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerDied","Data":"72d668ddd74f366fafe61d057c42cd41da6c58a021e9a9a3767056b0ebd96685"}
Nov 01 22:30:17.731604 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.731533    5845 scope.go:122] "RemoveContainer" containerID="72d668ddd74f366fafe61d057c42cd41da6c58a021e9a9a3767056b0ebd96685"
Nov 01 22:30:17.737342 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:17.732003    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892870-tn7rj_odh-ai-gateway-infra(97b88531-bad5-465a-8275-01770308a874)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" podUID="97b88531-bad5-465a-8275-01770308a874"
Nov 01 22:30:17.741314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.739344    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:17.753711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.753686    5845 scope.go:122] "RemoveContainer" containerID="c663360d0a2ac039453f09a72c1844bd29a03adea655d61186e25d756094118d"
Nov 01 22:30:17.854453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.854059    5845 scope.go:122] "RemoveContainer" containerID="5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971"
Nov 01 22:30:17.950964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:17.950939    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:18.078656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.078625    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\" (UniqueName: \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\") pod \"614512c6-d381-48ee-845c-fc429ba794b9\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") "
Nov 01 22:30:18.078886 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.078676    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\") pod \"614512c6-d381-48ee-845c-fc429ba794b9\" (UID: \"614512c6-d381-48ee-845c-fc429ba794b9\") "
Nov 01 22:30:18.079607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.079581    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca" pod "614512c6-d381-48ee-845c-fc429ba794b9" (UID: "614512c6-d381-48ee-845c-fc429ba794b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:18.081804 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.081782    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k" pod "614512c6-d381-48ee-845c-fc429ba794b9" (UID: "614512c6-d381-48ee-845c-fc429ba794b9"). InnerVolumeSpecName "kube-api-access-z6s8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:18.180200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.180168    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6s8k\" (UniqueName: \"kubernetes.io/projected/614512c6-d381-48ee-845c-fc429ba794b9-kube-api-access-z6s8k\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:18.180200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.180197    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/614512c6-d381-48ee-845c-fc429ba794b9-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:18.196193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196164    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:30:18.196846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196831    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196848    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196859    5845 cpu_manager.go:414] "containerMap: removing container" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196865    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196889    5845 cpu_manager.go:414] "containerMap: removing container" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196895    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.196901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196901    5845 cpu_manager.go:414] "containerMap: removing container" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196906    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196919    5845 cpu_manager.go:414] "containerMap: removing container" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196924    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196931    5845 cpu_manager.go:414] "containerMap: removing container" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196935    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196945    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196949    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196959    5845 cpu_manager.go:414] "containerMap: removing container" podUID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" containerName="payload-pre-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196963    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" containerName="payload-pre-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196984    5845 cpu_manager.go:414] "containerMap: removing container" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196990    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.196997    5845 cpu_manager.go:414] "containerMap: removing container" podUID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" containerName="payload-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197002    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" containerName="payload-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197089    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d7cba2-325a-48db-99da-27c4d5b27e90" containerName="payload-processing"
Nov 01 22:30:18.197096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197099    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197107    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2596176-d07d-49e5-b87f-b285e49045a6" containerName="istio-proxy"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197114    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfe9ea8-4374-4797-a4b6-fdaca5c675ba" containerName="payload-pre-processing"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197124    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c66cc6-05aa-449e-aa44-618406f83882" containerName="istio-proxy"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197130    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="664e03b1-bf13-4e9c-b606-ccea7580b2e0" containerName="payload-processing"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197137    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a76f1f-91d6-4c58-b228-bb7d44a660a3" containerName="payload-pre-processing"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197144    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.197585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.197154    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:30:18.201286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.201270    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.207199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.207164    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60807f-dockercfg-pt7s5\""
Nov 01 22:30:18.207199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.207166    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60807f\""
Nov 01 22:30:18.210969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.210947    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:30:18.409437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.399516    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mr75\" (UniqueName: \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.409437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.399854    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.501039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.500999    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.501236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.501115    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mr75\" (UniqueName: \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.501785 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.501764    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.513108 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.513089    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mr75\" (UniqueName: \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\") pod \"payload-processing-e2e-ipp-b-60807f-7674544b74-648g7\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.517979 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.517960    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:18.739570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.739542    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"
Nov 01 22:30:18.739570 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.739565    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5" event={"ID":"614512c6-d381-48ee-845c-fc429ba794b9","Type":"ContainerDied","Data":"15eb555a0d8c292f3e049dbe784ab0f1c5a45cd938b9b97dfc2ff7539c65fb4f"}
Nov 01 22:30:18.740190 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.739629    5845 scope.go:122] "RemoveContainer" containerID="dcb6e48a362410304127c909f53648ab700669726e570552cc0c42cca78ac6e8"
Nov 01 22:30:18.743776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.743754    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerStarted","Data":"ab35fd678cc182c2c7a5328fbd9abda478f40217e66ae573c947b634712b715c"}
Nov 01 22:30:18.797285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.797212    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" podStartSLOduration=18.797191324 podStartE2EDuration="18.797191324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:18.794120821 +0000 UTC m=+2722.050300587" watchObservedRunningTime="2026-11-01 22:30:18.797191324 +0000 UTC m=+2722.053371088"
Nov 01 22:30:18.805250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.805221    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:30:18.806757 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:30:18.806727    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18231c84_9262_4ab2_949e_968665edcbf1.slice/crio-b5cacca0a7f41bc9fcdad55b6e75310a41c4892b596b59bc66cae8e8efe12250 WatchSource:0}: Error finding container b5cacca0a7f41bc9fcdad55b6e75310a41c4892b596b59bc66cae8e8efe12250: Status 404 returned error can't find the container with id b5cacca0a7f41bc9fcdad55b6e75310a41c4892b596b59bc66cae8e8efe12250
Nov 01 22:30:18.812094 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.812072    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:18.816689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.816612    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-60807f-29892870-479x5"]
Nov 01 22:30:18.873591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.867738    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614512c6-d381-48ee-845c-fc429ba794b9" path="/var/lib/kubelet/pods/614512c6-d381-48ee-845c-fc429ba794b9/volumes"
Nov 01 22:30:18.945447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:18.945425    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:19.021186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.021095    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\" (UniqueName: \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\") pod \"97b88531-bad5-465a-8275-01770308a874\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") "
Nov 01 22:30:19.021186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.021156    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\") pod \"97b88531-bad5-465a-8275-01770308a874\" (UID: \"97b88531-bad5-465a-8275-01770308a874\") "
Nov 01 22:30:19.021986 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.021951    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca" pod "97b88531-bad5-465a-8275-01770308a874" (UID: "97b88531-bad5-465a-8275-01770308a874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:19.024707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.024680    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm" pod "97b88531-bad5-465a-8275-01770308a874" (UID: "97b88531-bad5-465a-8275-01770308a874"). InnerVolumeSpecName "kube-api-access-ghjdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:19.123167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.123129    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjdm\" (UniqueName: \"kubernetes.io/projected/97b88531-bad5-465a-8275-01770308a874-kube-api-access-ghjdm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:19.123167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.123166    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b88531-bad5-465a-8275-01770308a874-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:19.754631 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.754595    5845 generic.go:356] "Generic (PLEG): container finished" podID="a05e232c-d130-4b79-a95a-009bc1492770" containerID="ab35fd678cc182c2c7a5328fbd9abda478f40217e66ae573c947b634712b715c" exitCode=60
Nov 01 22:30:19.755093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.754679    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerDied","Data":"ab35fd678cc182c2c7a5328fbd9abda478f40217e66ae573c947b634712b715c"}
Nov 01 22:30:19.755093 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.754795    5845 scope.go:122] "RemoveContainer" containerID="5322ba051c5bb54d945c44d7aa55fa07e14743d004f0c51016a5245fddadf971"
Nov 01 22:30:19.755243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.755188    5845 scope.go:122] "RemoveContainer" containerID="ab35fd678cc182c2c7a5328fbd9abda478f40217e66ae573c947b634712b715c"
Nov 01 22:30:19.755552 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:19.755522    5845 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn_odh-ai-gateway-infra(a05e232c-d130-4b79-a95a-009bc1492770)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" podUID="a05e232c-d130-4b79-a95a-009bc1492770"
Nov 01 22:30:19.757793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.757732    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" event={"ID":"18231c84-9262-4ab2-949e-968665edcbf1","Type":"ContainerStarted","Data":"8abcfa7dbd8d3361dcc775ce83ac652d9b7a5b10a34e45079da58b214ee86b45"}
Nov 01 22:30:19.757919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.757817    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" event={"ID":"18231c84-9262-4ab2-949e-968665edcbf1","Type":"ContainerStarted","Data":"b5cacca0a7f41bc9fcdad55b6e75310a41c4892b596b59bc66cae8e8efe12250"}
Nov 01 22:30:19.757919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.757849    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:19.759402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.759385    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj" event={"ID":"97b88531-bad5-465a-8275-01770308a874","Type":"ContainerDied","Data":"984f24c06500a1d8d4a2f950734c5fe10c10ebbe6409b4ce5846a9c0e8364f13"}
Nov 01 22:30:19.759501 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.759425    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"
Nov 01 22:30:19.777481 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.770356    5845 scope.go:122] "RemoveContainer" containerID="72d668ddd74f366fafe61d057c42cd41da6c58a021e9a9a3767056b0ebd96685"
Nov 01 22:30:19.792069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.792021    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" podStartSLOduration=1.79200684 podStartE2EDuration="1.79200684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:19.791419575 +0000 UTC m=+2723.047599340" watchObservedRunningTime="2026-11-01 22:30:19.79200684 +0000 UTC m=+2723.048186604"
Nov 01 22:30:19.809608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.809582    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:19.814309 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.814275    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:19.821845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.821825    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892870-tn7rj"]
Nov 01 22:30:19.895680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.895633    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:30:19.895914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:19.895886    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" containerID="cri-o://07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7" gracePeriod=30
Nov 01 22:30:20.868904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:20.868872    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b88531-bad5-465a-8275-01770308a874" path="/var/lib/kubelet/pods/97b88531-bad5-465a-8275-01770308a874/volumes"
Nov 01 22:30:20.958344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:20.958285    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:21.054885 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.054789    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\") pod \"a05e232c-d130-4b79-a95a-009bc1492770\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") "
Nov 01 22:30:21.055077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.054933    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\" (UniqueName: \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\") pod \"a05e232c-d130-4b79-a95a-009bc1492770\" (UID: \"a05e232c-d130-4b79-a95a-009bc1492770\") "
Nov 01 22:30:21.055227 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.055204    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca" pod "a05e232c-d130-4b79-a95a-009bc1492770" (UID: "a05e232c-d130-4b79-a95a-009bc1492770"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:21.055807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.055781    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05e232c-d130-4b79-a95a-009bc1492770-maas-api-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:21.057667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.057640    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f" pod "a05e232c-d130-4b79-a95a-009bc1492770" (UID: "a05e232c-d130-4b79-a95a-009bc1492770"). InnerVolumeSpecName "kube-api-access-sd56f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:21.156716 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.156683    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd56f\" (UniqueName: \"kubernetes.io/projected/a05e232c-d130-4b79-a95a-009bc1492770-kube-api-access-sd56f\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:21.515843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.515801    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing" containerID="cri-o://f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e" gracePeriod=25
Nov 01 22:30:21.592921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.592879    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:30:21.592921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.592951    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:30:21.792564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.792484    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn" event={"ID":"a05e232c-d130-4b79-a95a-009bc1492770","Type":"ContainerDied","Data":"6da839c9d4c285989a31ce599359eb21c0fd0c89d1eba617cc9c26acc97d6f62"}
Nov 01 22:30:21.792564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.792557    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"
Nov 01 22:30:21.792796 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.792582    5845 scope.go:122] "RemoveContainer" containerID="ab35fd678cc182c2c7a5328fbd9abda478f40217e66ae573c947b634712b715c"
Nov 01 22:30:21.817826 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.817802    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:21.824043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:21.824018    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-56452a-29892870-92bsn"]
Nov 01 22:30:22.138173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:22.138087    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:30:22.863080 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:22.863043    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a05e232c-d130-4b79-a95a-009bc1492770" path="/var/lib/kubelet/pods/a05e232c-d130-4b79-a95a-009bc1492770/volumes"
Nov 01 22:30:25.180970 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.180912    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:30:25.210492 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210467    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210506    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210583    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210616    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210651    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210742    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210808    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.210914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210869    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.211067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.210922    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\") pod \"36470550-abf4-47f8-8c5e-583fc7e875d5\" (UID: \"36470550-abf4-47f8-8c5e-583fc7e875d5\") "
Nov 01 22:30:25.211067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211046    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:25.211991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211343    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:25.211991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211678    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:25.211991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211928    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36470550-abf4-47f8-8c5e-583fc7e875d5-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.211991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211950    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.211991 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.211963    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.212903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.212864    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:25.213148 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.213114    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:25.213900 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.213860    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:25.214020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.213983    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:25.221155 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.221127    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:30:25.222551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.222522    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5" pod "36470550-abf4-47f8-8c5e-583fc7e875d5" (UID: "36470550-abf4-47f8-8c5e-583fc7e875d5"). InnerVolumeSpecName "kube-api-access-ml9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:25.276698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.276495    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 22:30:25.296331 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.296284    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 22:30:25.313557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313524    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.313557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313556    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.313811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313572    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.313811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313588    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.313811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313603    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml9k5\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-kube-api-access-ml9k5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.313811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.313618    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36470550-abf4-47f8-8c5e-583fc7e875d5-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:25.836981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.836950    5845 generic.go:356] "Generic (PLEG): container finished" podID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerID="07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7" exitCode=0
Nov 01 22:30:25.837173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.837003    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"
Nov 01 22:30:25.837173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.837037    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerDied","Data":"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"}
Nov 01 22:30:25.837173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.837092    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj" event={"ID":"36470550-abf4-47f8-8c5e-583fc7e875d5","Type":"ContainerDied","Data":"1001036b2111b505314c601d94e0d539acc76d585c53adc8c5d52eccdc632743"}
Nov 01 22:30:25.837173 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.837114    5845 scope.go:122] "RemoveContainer" containerID="07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"
Nov 01 22:30:25.848939 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.848922    5845 scope.go:122] "RemoveContainer" containerID="037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"
Nov 01 22:30:25.859903 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.859803    5845 scope.go:122] "RemoveContainer" containerID="07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"
Nov 01 22:30:25.860158 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:25.860131    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7\": container with ID starting with 07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7 not found: ID does not exist" containerID="07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"
Nov 01 22:30:25.860257 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.860170    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7"} err="failed to get container status \"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7\": rpc error: code = NotFound desc = could not find container \"07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7\": container with ID starting with 07bb807ab7c7ed86e6d27e6175b58c4944379f1ec9c2cff7b30f1766fa52bff7 not found: ID does not exist"
Nov 01 22:30:25.860257 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.860192    5845 scope.go:122] "RemoveContainer" containerID="037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"
Nov 01 22:30:25.860576 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:25.860555    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7\": container with ID starting with 037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7 not found: ID does not exist" containerID="037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"
Nov 01 22:30:25.860649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.860582    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7"} err="failed to get container status \"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7\": rpc error: code = NotFound desc = could not find container \"037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7\": container with ID starting with 037c652c77ff65e4a7cbf0752dd1c07eb16ea5ffa6bfaef2c3c8ff1b42ffb4b7 not found: ID does not exist"
Nov 01 22:30:25.862239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.862217    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:30:25.867884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:25.867863    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-51e3b7-openshift-default-b5dc5b5cc-t5rxj"]
Nov 01 22:30:26.021371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.021136    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:30:26.021559 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.021435    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api" containerID="cri-o://d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b" gracePeriod=30
Nov 01 22:30:26.419268 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.419237    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:30:26.556494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.556461    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\") pod \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") "
Nov 01 22:30:26.556672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.556551    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\" (UniqueName: \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\") pod \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") "
Nov 01 22:30:26.556672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.556598    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\") pod \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") "
Nov 01 22:30:26.556843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.556815    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca" pod "2db1449e-00fb-409c-8a1f-2a06ad3f94d5" (UID: "2db1449e-00fb-409c-8a1f-2a06ad3f94d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:26.556893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.556831    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\") pod \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\" (UID: \"2db1449e-00fb-409c-8a1f-2a06ad3f94d5\") "
Nov 01 22:30:26.557548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.557521    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:26.558730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.558707    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls" pod "2db1449e-00fb-409c-8a1f-2a06ad3f94d5" (UID: "2db1449e-00fb-409c-8a1f-2a06ad3f94d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:26.558829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.558789    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs" pod "2db1449e-00fb-409c-8a1f-2a06ad3f94d5" (UID: "2db1449e-00fb-409c-8a1f-2a06ad3f94d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:26.558829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.558801    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw" pod "2db1449e-00fb-409c-8a1f-2a06ad3f94d5" (UID: "2db1449e-00fb-409c-8a1f-2a06ad3f94d5"). InnerVolumeSpecName "kube-api-access-p6hgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:26.659112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.659026    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6hgw\" (UniqueName: \"kubernetes.io/projected/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-kube-api-access-p6hgw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:26.659112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.659055    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:26.659112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.659071    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2db1449e-00fb-409c-8a1f-2a06ad3f94d5-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:26.845494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.845464    5845 generic.go:356] "Generic (PLEG): container finished" podID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerID="d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b" exitCode=0
Nov 01 22:30:26.845656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.845510    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"
Nov 01 22:30:26.845656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.845548    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" event={"ID":"2db1449e-00fb-409c-8a1f-2a06ad3f94d5","Type":"ContainerDied","Data":"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"}
Nov 01 22:30:26.845656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.845595    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk" event={"ID":"2db1449e-00fb-409c-8a1f-2a06ad3f94d5","Type":"ContainerDied","Data":"fe3f4a7403c8f7596d4276fa47cb7d7119449b117b727df3050533396ec3bbf1"}
Nov 01 22:30:26.845656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.845622    5845 scope.go:122] "RemoveContainer" containerID="d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"
Nov 01 22:30:26.863016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.859348    5845 scope.go:122] "RemoveContainer" containerID="d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"
Nov 01 22:30:26.863016 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:26.859784    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b\": container with ID starting with d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b not found: ID does not exist" containerID="d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"
Nov 01 22:30:26.863016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.859820    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b"} err="failed to get container status \"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b\": rpc error: code = NotFound desc = could not find container \"d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b\": container with ID starting with d3569cc8fe957df1aa06f9256759e71b83ad1344d5e005bc5a1991351dba2e3b not found: ID does not exist"
Nov 01 22:30:26.875041 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.875014    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" path="/var/lib/kubelet/pods/36470550-abf4-47f8-8c5e-583fc7e875d5/volumes"
Nov 01 22:30:26.880157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.880132    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:30:26.885162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:26.885138    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-f758d9-558bd59bfb-zgtmk"]
Nov 01 22:30:28.862514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:28.862481    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" path="/var/lib/kubelet/pods/2db1449e-00fb-409c-8a1f-2a06ad3f94d5/volumes"
Nov 01 22:30:30.787486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:30.787456    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:30:31.023321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:31.016171    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:30:32.137199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:32.137155    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:30:33.795103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:33.795070    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:30:33.795607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:33.795305    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" containerID="cri-o://fce0365b7df0c2241ce80f02c638dfae77aa9a3c16e39eb5b7acaa791cc2f11d" gracePeriod=30
Nov 01 22:30:34.819391 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:34.819352    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:30:35.393083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.393052    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:30:35.585618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.585575    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\" (UniqueName: \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\") pod \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") "
Nov 01 22:30:35.585791 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.585716    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\") pod \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\" (UID: \"f123a7c8-5b58-4630-8eb1-4984e30e9cfb\") "
Nov 01 22:30:35.586164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.586126    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume" pod "f123a7c8-5b58-4630-8eb1-4984e30e9cfb" (UID: "f123a7c8-5b58-4630-8eb1-4984e30e9cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:35.586461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.586438    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:35.588894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.588866    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v" pod "f123a7c8-5b58-4630-8eb1-4984e30e9cfb" (UID: "f123a7c8-5b58-4630-8eb1-4984e30e9cfb"). InnerVolumeSpecName "kube-api-access-4n89v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:35.687413 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.687322    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n89v\" (UniqueName: \"kubernetes.io/projected/f123a7c8-5b58-4630-8eb1-4984e30e9cfb-kube-api-access-4n89v\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:35.916030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.915997    5845 generic.go:356] "Generic (PLEG): container finished" podID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerID="69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b" exitCode=0
Nov 01 22:30:35.916513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.916042    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"
Nov 01 22:30:35.916513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.916094    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" event={"ID":"f123a7c8-5b58-4630-8eb1-4984e30e9cfb","Type":"ContainerDied","Data":"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"}
Nov 01 22:30:35.916513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.916160    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h" event={"ID":"f123a7c8-5b58-4630-8eb1-4984e30e9cfb","Type":"ContainerDied","Data":"ab151241ac2f52928c6cd224d838867bac7bc53b1e3bd533f4d447c0cf8a0ab2"}
Nov 01 22:30:35.916513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.916195    5845 scope.go:122] "RemoveContainer" containerID="69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"
Nov 01 22:30:35.928083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.928065    5845 scope.go:122] "RemoveContainer" containerID="69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"
Nov 01 22:30:35.928493 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:35.928473    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b\": container with ID starting with 69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b not found: ID does not exist" containerID="69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"
Nov 01 22:30:35.928553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.928502    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b"} err="failed to get container status \"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b\": rpc error: code = NotFound desc = could not find container \"69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b\": container with ID starting with 69c849af66b81de637ba31a2821d318b7037d4ebe54cc6a717b28ca99666987b not found: ID does not exist"
Nov 01 22:30:35.939959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.939897    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:30:35.944188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:35.944161    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-51e3b7-bbcf88cf-zrf8h"]
Nov 01 22:30:36.731700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:36.731659    5845 patch_prober.go:29] interesting pod/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:30:36.731910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:36.731740    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:30:36.863280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:36.863249    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" path="/var/lib/kubelet/pods/f123a7c8-5b58-4630-8eb1-4984e30e9cfb/volumes"
Nov 01 22:30:38.954992 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:38.954957    5845 generic.go:356] "Generic (PLEG): container finished" podID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerID="fce0365b7df0c2241ce80f02c638dfae77aa9a3c16e39eb5b7acaa791cc2f11d" exitCode=0
Nov 01 22:30:38.955438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:38.955025    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerDied","Data":"fce0365b7df0c2241ce80f02c638dfae77aa9a3c16e39eb5b7acaa791cc2f11d"}
Nov 01 22:30:38.955438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:38.955088    5845 scope.go:122] "RemoveContainer" containerID="f484bfa3b61df0e2f9dd8eeac4379b86b596c53045c86708874d4c752c1d4564"
Nov 01 22:30:39.092927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.092905    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:30:39.135489 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135455    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.135690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135532    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.135690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135629    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.135690 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135675    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.135909 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135813    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:39.135909 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135841    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.136030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135926    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.136030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.135966    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.136030 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.136000    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.136180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.136033    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\") pod \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\" (UID: \"4042e0fa-32fa-46cc-8d90-7b13742efcf5\") "
Nov 01 22:30:39.136964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.136689    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.136964 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.136919    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:39.137140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.137114    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:39.142377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142290    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:39.142377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142340    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:39.142630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142392    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "kube-api-access-8xhnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:39.142630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142491    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:39.142630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142536    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:30:39.142630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.142601    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert" pod "4042e0fa-32fa-46cc-8d90-7b13742efcf5" (UID: "4042e0fa-32fa-46cc-8d90-7b13742efcf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:39.238113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238088    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238114    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238125    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238135    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238143    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238153    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xhnt\" (UniqueName: \"kubernetes.io/projected/4042e0fa-32fa-46cc-8d90-7b13742efcf5-kube-api-access-8xhnt\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238162    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.238260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.238171    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4042e0fa-32fa-46cc-8d90-7b13742efcf5-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:39.963472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.963440    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw" event={"ID":"4042e0fa-32fa-46cc-8d90-7b13742efcf5","Type":"ContainerDied","Data":"e9042e8634157e4b01857026437fe0f552d293823df81ef7bd71ccd85265c52a"}
Nov 01 22:30:39.963472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.963458    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"
Nov 01 22:30:39.963996 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.963496    5845 scope.go:122] "RemoveContainer" containerID="fce0365b7df0c2241ce80f02c638dfae77aa9a3c16e39eb5b7acaa791cc2f11d"
Nov 01 22:30:39.987597 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.987574    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:30:39.993982 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:39.993956    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-f758d9-openshift-default-765d9d8f7f-x4zkw"]
Nov 01 22:30:40.300138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.300089    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main" containerID="cri-o://02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88" gracePeriod=45
Nov 01 22:30:40.325748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.325709    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" containerID="cri-o://b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b" gracePeriod=45
Nov 01 22:30:40.659543 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.659511    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:30:40.664129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.664107    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:30:40.758281 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758249    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.758456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758332    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.758456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758362    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.758456 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758417    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.758616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758468    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.758616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758572    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\" (UniqueName: \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.758738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758624    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.758738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758676    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.758738 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758727    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.758882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758763    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.758882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758773    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.758968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758882    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\" (UniqueName: \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.758968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758920    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\" (UniqueName: \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.758968 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.758950    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\") pod \"62b0fab0-718a-497e-881d-70a837997119\" (UID: \"62b0fab0-718a-497e-881d-70a837997119\") "
Nov 01 22:30:40.759101 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759000    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\") pod \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\" (UID: \"4cb43ab1-4a4a-4d10-8245-7c0d275d2813\") "
Nov 01 22:30:40.759380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759358    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.759611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759571    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.759611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759600    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.759811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759621    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.759811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759630    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.759811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759670    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.759811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.759687    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.761071 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761039    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:40.761711 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761686    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.761797 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761758    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "kube-api-access-484dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:40.761845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761826    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:30:40.761882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761844    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs" pod "62b0fab0-718a-497e-881d-70a837997119" (UID: "62b0fab0-718a-497e-881d-70a837997119"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:40.761882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.761863    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx" pod "4cb43ab1-4a4a-4d10-8245-7c0d275d2813" (UID: "4cb43ab1-4a4a-4d10-8245-7c0d275d2813"). InnerVolumeSpecName "kube-api-access-d9pvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:40.860482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860452    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860480    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pvx\" (UniqueName: \"kubernetes.io/projected/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-kube-api-access-d9pvx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860482 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860490    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860499    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860508    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484dt\" (UniqueName: \"kubernetes.io/projected/62b0fab0-718a-497e-881d-70a837997119-kube-api-access-484dt\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860518    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62b0fab0-718a-497e-881d-70a837997119-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860526    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62b0fab0-718a-497e-881d-70a837997119-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.860693 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.860534    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb43ab1-4a4a-4d10-8245-7c0d275d2813-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:40.862852 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.862826    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" path="/var/lib/kubelet/pods/4042e0fa-32fa-46cc-8d90-7b13742efcf5/volumes"
Nov 01 22:30:40.971692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.971634    5845 generic.go:356] "Generic (PLEG): container finished" podID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerID="02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88" exitCode=0
Nov 01 22:30:40.971692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.971686    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"
Nov 01 22:30:40.972213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.971729    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" event={"ID":"4cb43ab1-4a4a-4d10-8245-7c0d275d2813","Type":"ContainerDied","Data":"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"}
Nov 01 22:30:40.972213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.971774    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz" event={"ID":"4cb43ab1-4a4a-4d10-8245-7c0d275d2813","Type":"ContainerDied","Data":"e660a1ddc562772668d1fab9fcea64f44cdd06f0087d5ebb308e1aa1fae4bcde"}
Nov 01 22:30:40.972213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.971792    5845 scope.go:122] "RemoveContainer" containerID="02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"
Nov 01 22:30:40.974974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.974933    5845 generic.go:356] "Generic (PLEG): container finished" podID="62b0fab0-718a-497e-881d-70a837997119" containerID="b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b" exitCode=0
Nov 01 22:30:40.975103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.974975    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"
Nov 01 22:30:40.975103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.975026    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" event={"ID":"62b0fab0-718a-497e-881d-70a837997119","Type":"ContainerDied","Data":"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"}
Nov 01 22:30:40.975103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.975070    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" event={"ID":"62b0fab0-718a-497e-881d-70a837997119","Type":"ContainerDied","Data":"0415e61ef19e47dc09aad62f74a7e23809f3a926ead2c894280336b0e6c9c590"}
Nov 01 22:30:40.983679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.983656    5845 scope.go:122] "RemoveContainer" containerID="02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"
Nov 01 22:30:40.983937 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:40.983919    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88\": container with ID starting with 02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88 not found: ID does not exist" containerID="02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"
Nov 01 22:30:40.984013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.983945    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88"} err="failed to get container status \"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88\": rpc error: code = NotFound desc = could not find container \"02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88\": container with ID starting with 02f1fa05dce460f54a7f9ff783ba773844869b6ed2c4db55050b490ade5f6f88 not found: ID does not exist"
Nov 01 22:30:40.984013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.983960    5845 scope.go:122] "RemoveContainer" containerID="b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"
Nov 01 22:30:40.993764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.993663    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 22:30:40.993828 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.993801    5845 scope.go:122] "RemoveContainer" containerID="b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"
Nov 01 22:30:40.994087 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:40.994065    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b\": container with ID starting with b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b not found: ID does not exist" containerID="b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"
Nov 01 22:30:40.994127 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:40.994094    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b"} err="failed to get container status \"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b\": rpc error: code = NotFound desc = could not find container \"b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b\": container with ID starting with b961c7de0d6fdfdf2104b4c2b46dcac3fb412c8e6b1dc48a70627d59899ff94b not found: ID does not exist"
Nov 01 22:30:41.003235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.003212    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:30:41.008645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.008625    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7"]
Nov 01 22:30:41.021001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.020980    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 22:30:41.028078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.028049    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-53b784/distinct-w1-53b784-kserve-5d545c4ff9-qb8zz"]
Nov 01 22:30:41.587092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.587054    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-53b784/simulator-w1-53b784-kserve-76d55f9477-pl9k7" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:30:41.913682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:41.913662    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:30:42.167374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.166930    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\") pod \"c2b65487-1cd9-4053-bb63-f50c185beb79\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") "
Nov 01 22:30:42.167374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.167078    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\" (UniqueName: \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\") pod \"c2b65487-1cd9-4053-bb63-f50c185beb79\" (UID: \"c2b65487-1cd9-4053-bb63-f50c185beb79\") "
Nov 01 22:30:42.204356 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.203699    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume" pod "c2b65487-1cd9-4053-bb63-f50c185beb79" (UID: "c2b65487-1cd9-4053-bb63-f50c185beb79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:30:42.304327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.270822    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b65487-1cd9-4053-bb63-f50c185beb79-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:42.333092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.332743    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv" pod "c2b65487-1cd9-4053-bb63-f50c185beb79" (UID: "c2b65487-1cd9-4053-bb63-f50c185beb79"). InnerVolumeSpecName "kube-api-access-bj5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:42.344901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.344867    5845 generic.go:356] "Generic (PLEG): container finished" podID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerID="f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e" exitCode=0
Nov 01 22:30:42.345153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.344942    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" event={"ID":"c2b65487-1cd9-4053-bb63-f50c185beb79","Type":"ContainerDied","Data":"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"}
Nov 01 22:30:42.345343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.345003    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"
Nov 01 22:30:42.345527 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.345503    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc" event={"ID":"c2b65487-1cd9-4053-bb63-f50c185beb79","Type":"ContainerDied","Data":"c29dc2351ebd9da5587bd4d7bf5833ca26f68218b5cea8eff35405bdfe42d9bf"}
Nov 01 22:30:42.350555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.349687    5845 scope.go:122] "RemoveContainer" containerID="f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"
Nov 01 22:30:42.369720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.369701    5845 scope.go:122] "RemoveContainer" containerID="f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"
Nov 01 22:30:42.371690 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:30:42.370121    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e\": container with ID starting with f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e not found: ID does not exist" containerID="f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"
Nov 01 22:30:42.371847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.370154    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e"} err="failed to get container status \"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e\": rpc error: code = NotFound desc = could not find container \"f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e\": container with ID starting with f8179dad3e41d43d7fae7f5250d7a28e3c6d4dcb08eeb1161692c64865c64c9e not found: ID does not exist"
Nov 01 22:30:42.381743 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.381678    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:30:42.388333 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.385913    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-f758d9-67b9b75799-fvfsc"]
Nov 01 22:30:42.410321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.406384    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5kv\" (UniqueName: \"kubernetes.io/projected/c2b65487-1cd9-4053-bb63-f50c185beb79-kube-api-access-bj5kv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:42.862407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.862378    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" path="/var/lib/kubelet/pods/4cb43ab1-4a4a-4d10-8245-7c0d275d2813/volumes"
Nov 01 22:30:42.862953 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.862928    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b0fab0-718a-497e-881d-70a837997119" path="/var/lib/kubelet/pods/62b0fab0-718a-497e-881d-70a837997119/volumes"
Nov 01 22:30:42.863511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:42.863487    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" path="/var/lib/kubelet/pods/c2b65487-1cd9-4053-bb63-f50c185beb79/volumes"
Nov 01 22:30:44.339059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:44.339021    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:30:47.884955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:47.884623    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n_07d36006-6cbe-4f59-aac1-5b1302e5beb2/payload-processing/0.log"
Nov 01 22:30:48.494384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:48.494356    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr_e07fa865-0c48-457d-8dc7-fd430c157021/payload-processing/0.log"
Nov 01 22:30:48.659314 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:48.659274    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr_e07fa865-0c48-457d-8dc7-fd430c157021/payload-processing/0.log"
Nov 01 22:30:49.339851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:49.339790    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing" containerID="cri-o://c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c" gracePeriod=25
Nov 01 22:30:56.317364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:30:56.317287    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:00.061635 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:00.061589    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 22:31:00.081942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:00.081905    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 22:31:01.505609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:01.503259    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 22:31:01.505609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:01.503896    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy" containerID="cri-o://c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9" gracePeriod=30
Nov 01 22:31:04.815569 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:04.815525    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:31:06.316698 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.316662    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:06.777920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.777895    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 22:31:06.907878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.907792    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.907878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.907835    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.907925    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908135 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.907952    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908140    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908153    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:06.908261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908194    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908243    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908268    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908341    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908400    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908427    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908428    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\") pod \"ae7bbaad-d051-4782-a918-386d7f69bd6d\" (UID: \"ae7bbaad-d051-4782-a918-386d7f69bd6d\") "
Nov 01 22:31:06.908545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:06.908966 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908938    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:06.909720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908968    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7bbaad-d051-4782-a918-386d7f69bd6d-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:06.909720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908985    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:06.909720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.908999    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:06.909720 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.909013    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:06.910800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.910775    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:06.910919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.910901    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:06.911196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.911171    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:06.911286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:06.911199    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4" pod "ae7bbaad-d051-4782-a918-386d7f69bd6d" (UID: "ae7bbaad-d051-4782-a918-386d7f69bd6d"). InnerVolumeSpecName "kube-api-access-sj6k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:07.010557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.010445    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:07.010557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.010483    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6k4\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-kube-api-access-sj6k4\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:07.010557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.010498    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:07.010557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.010514    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7bbaad-d051-4782-a918-386d7f69bd6d-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:07.563923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.563886    5845 generic.go:356] "Generic (PLEG): container finished" podID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerID="c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9" exitCode=0
Nov 01 22:31:07.564374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.563984    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"
Nov 01 22:31:07.564374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.564017    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" event={"ID":"ae7bbaad-d051-4782-a918-386d7f69bd6d","Type":"ContainerDied","Data":"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"}
Nov 01 22:31:07.564374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.564074    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9" event={"ID":"ae7bbaad-d051-4782-a918-386d7f69bd6d","Type":"ContainerDied","Data":"c70fc7b1fe6a0eda448f60cda77446a0fbb178d3e908cb71125b2964aa5c26b1"}
Nov 01 22:31:07.564374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.564103    5845 scope.go:122] "RemoveContainer" containerID="c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"
Nov 01 22:31:07.578957 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.578936    5845 scope.go:122] "RemoveContainer" containerID="c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"
Nov 01 22:31:07.579373 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:07.579342    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9\": container with ID starting with c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9 not found: ID does not exist" containerID="c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"
Nov 01 22:31:07.579464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.579383    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9"} err="failed to get container status \"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9\": rpc error: code = NotFound desc = could not find container \"c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9\": container with ID starting with c2203cddedf406103ecf6ecdc1c1ded5608976e97f7f7c5e0ffbd8f09289e0a9 not found: ID does not exist"
Nov 01 22:31:07.591279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.591257    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 22:31:07.599636 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:07.599611    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-53b784-openshift-default-7d967d7c57-2m7b9"]
Nov 01 22:31:08.863841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:08.863801    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" path="/var/lib/kubelet/pods/ae7bbaad-d051-4782-a918-386d7f69bd6d/volumes"
Nov 01 22:31:09.401278 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:09.401251    5845 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039aafea_77e9_4db4_9e2c_faf5abf0779f.slice/crio-c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 22:31:09.580917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.580888    5845 generic.go:356] "Generic (PLEG): container finished" podID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerID="c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c" exitCode=0
Nov 01 22:31:09.581069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.580961    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" event={"ID":"039aafea-77e9-4db4-9e2c-faf5abf0779f","Type":"ContainerDied","Data":"c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c"}
Nov 01 22:31:09.624643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.624622    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:31:09.746712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.746670    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\" (UniqueName: \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\") pod \"039aafea-77e9-4db4-9e2c-faf5abf0779f\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") "
Nov 01 22:31:09.746899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.746733    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\") pod \"039aafea-77e9-4db4-9e2c-faf5abf0779f\" (UID: \"039aafea-77e9-4db4-9e2c-faf5abf0779f\") "
Nov 01 22:31:09.747159 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.747129    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume" pod "039aafea-77e9-4db4-9e2c-faf5abf0779f" (UID: "039aafea-77e9-4db4-9e2c-faf5abf0779f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:09.749359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.749334    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9" pod "039aafea-77e9-4db4-9e2c-faf5abf0779f" (UID: "039aafea-77e9-4db4-9e2c-faf5abf0779f"). InnerVolumeSpecName "kube-api-access-bmpc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:09.848199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.848164    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmpc9\" (UniqueName: \"kubernetes.io/projected/039aafea-77e9-4db4-9e2c-faf5abf0779f-kube-api-access-bmpc9\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:09.848199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:09.848195    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/039aafea-77e9-4db4-9e2c-faf5abf0779f-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:10.589998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.589954    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"
Nov 01 22:31:10.590524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.589996    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj" event={"ID":"039aafea-77e9-4db4-9e2c-faf5abf0779f","Type":"ContainerDied","Data":"7cdb1866d9fea11531a1a0a4eee1a013f6f2cf1acf452095fa7610d06eb84655"}
Nov 01 22:31:10.590524 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.590062    5845 scope.go:122] "RemoveContainer" containerID="c13e601755229cd24e5cfef867b5baf6e0779551e3081f84e973e50df841cb1c"
Nov 01 22:31:10.616152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.616130    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:31:10.619927 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.619903    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-2c5135-7b9d8c46c4-kmfjj"]
Nov 01 22:31:10.861998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:10.861923    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" path="/var/lib/kubelet/pods/039aafea-77e9-4db4-9e2c-faf5abf0779f/volumes"
Nov 01 22:31:13.421692 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.421653    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:31:13.422271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.421895    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" podUID="1dfdb033-56bf-41e6-93df-e8a66e3de202" containerName="maas-api" containerID="cri-o://2bd7c323d3a631c5944bda084fba0b7d7989d5b10a5cf3d46be075c8305417f7" gracePeriod=30
Nov 01 22:31:13.483942 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.483911    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:31:13.484187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.484155    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" podUID="18231c84-9262-4ab2-949e-968665edcbf1" containerName="payload-processing" containerID="cri-o://8abcfa7dbd8d3361dcc775ce83ac652d9b7a5b10a34e45079da58b214ee86b45" gracePeriod=30
Nov 01 22:31:13.626389 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.626325    5845 generic.go:356] "Generic (PLEG): container finished" podID="1dfdb033-56bf-41e6-93df-e8a66e3de202" containerID="2bd7c323d3a631c5944bda084fba0b7d7989d5b10a5cf3d46be075c8305417f7" exitCode=0
Nov 01 22:31:13.626577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.626465    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" event={"ID":"1dfdb033-56bf-41e6-93df-e8a66e3de202","Type":"ContainerDied","Data":"2bd7c323d3a631c5944bda084fba0b7d7989d5b10a5cf3d46be075c8305417f7"}
Nov 01 22:31:13.629170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.629138    5845 generic.go:356] "Generic (PLEG): container finished" podID="18231c84-9262-4ab2-949e-968665edcbf1" containerID="8abcfa7dbd8d3361dcc775ce83ac652d9b7a5b10a34e45079da58b214ee86b45" exitCode=0
Nov 01 22:31:13.629480 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.629449    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" event={"ID":"18231c84-9262-4ab2-949e-968665edcbf1","Type":"ContainerDied","Data":"8abcfa7dbd8d3361dcc775ce83ac652d9b7a5b10a34e45079da58b214ee86b45"}
Nov 01 22:31:13.843571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.843542    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:31:13.911921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:13.911898    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:31:14.015723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.015694    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\" (UniqueName: \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\") pod \"1dfdb033-56bf-41e6-93df-e8a66e3de202\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") "
Nov 01 22:31:14.015897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.015810    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\") pod \"1dfdb033-56bf-41e6-93df-e8a66e3de202\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") "
Nov 01 22:31:14.015897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.015835    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\") pod \"1dfdb033-56bf-41e6-93df-e8a66e3de202\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") "
Nov 01 22:31:14.015990 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.015956    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\" (UniqueName: \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\") pod \"18231c84-9262-4ab2-949e-968665edcbf1\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") "
Nov 01 22:31:14.016077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016055    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\") pod \"1dfdb033-56bf-41e6-93df-e8a66e3de202\" (UID: \"1dfdb033-56bf-41e6-93df-e8a66e3de202\") "
Nov 01 22:31:14.016224 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016191    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\") pod \"18231c84-9262-4ab2-949e-968665edcbf1\" (UID: \"18231c84-9262-4ab2-949e-968665edcbf1\") "
Nov 01 22:31:14.016414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016233    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca" pod "1dfdb033-56bf-41e6-93df-e8a66e3de202" (UID: "1dfdb033-56bf-41e6-93df-e8a66e3de202"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:14.016697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016660    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume" pod "18231c84-9262-4ab2-949e-968665edcbf1" (UID: "18231c84-9262-4ab2-949e-968665edcbf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:14.016908 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016885    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dfdb033-56bf-41e6-93df-e8a66e3de202-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.016985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.016915    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18231c84-9262-4ab2-949e-968665edcbf1-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.018129 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.018100    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs" pod "1dfdb033-56bf-41e6-93df-e8a66e3de202" (UID: "1dfdb033-56bf-41e6-93df-e8a66e3de202"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:14.018221 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.018142    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls" pod "1dfdb033-56bf-41e6-93df-e8a66e3de202" (UID: "1dfdb033-56bf-41e6-93df-e8a66e3de202"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:14.018221 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.018141    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5" pod "1dfdb033-56bf-41e6-93df-e8a66e3de202" (UID: "1dfdb033-56bf-41e6-93df-e8a66e3de202"). InnerVolumeSpecName "kube-api-access-76hm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:14.018322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.018228    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75" pod "18231c84-9262-4ab2-949e-968665edcbf1" (UID: "18231c84-9262-4ab2-949e-968665edcbf1"). InnerVolumeSpecName "kube-api-access-2mr75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:14.118625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.118542    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mr75\" (UniqueName: \"kubernetes.io/projected/18231c84-9262-4ab2-949e-968665edcbf1-kube-api-access-2mr75\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.118625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.118577    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.118625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.118588    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hm5\" (UniqueName: \"kubernetes.io/projected/1dfdb033-56bf-41e6-93df-e8a66e3de202-kube-api-access-76hm5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.118625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.118597    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dfdb033-56bf-41e6-93df-e8a66e3de202-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:14.637422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.637392    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"
Nov 01 22:31:14.637422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.637399    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp" event={"ID":"1dfdb033-56bf-41e6-93df-e8a66e3de202","Type":"ContainerDied","Data":"311218d2239ce5320f04f208f58fcc74e81ae5a8d38a8371bb6b7110bafa4f7e"}
Nov 01 22:31:14.637943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.637468    5845 scope.go:122] "RemoveContainer" containerID="2bd7c323d3a631c5944bda084fba0b7d7989d5b10a5cf3d46be075c8305417f7"
Nov 01 22:31:14.639225 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.639203    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"
Nov 01 22:31:14.639367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.639237    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7" event={"ID":"18231c84-9262-4ab2-949e-968665edcbf1","Type":"ContainerDied","Data":"b5cacca0a7f41bc9fcdad55b6e75310a41c4892b596b59bc66cae8e8efe12250"}
Nov 01 22:31:14.649043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.648975    5845 scope.go:122] "RemoveContainer" containerID="8abcfa7dbd8d3361dcc775ce83ac652d9b7a5b10a34e45079da58b214ee86b45"
Nov 01 22:31:14.668794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.668771    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:31:14.672901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.672876    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60807f-667857dd76-86nbp"]
Nov 01 22:31:14.683234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.683212    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:31:14.688788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.688766    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60807f-7674544b74-648g7"]
Nov 01 22:31:14.862408 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.862366    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18231c84-9262-4ab2-949e-968665edcbf1" path="/var/lib/kubelet/pods/18231c84-9262-4ab2-949e-968665edcbf1/volumes"
Nov 01 22:31:14.862834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:14.862816    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfdb033-56bf-41e6-93df-e8a66e3de202" path="/var/lib/kubelet/pods/1dfdb033-56bf-41e6-93df-e8a66e3de202/volumes"
Nov 01 22:31:15.094982 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.094929    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main" containerID="cri-o://2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08" gracePeriod=45
Nov 01 22:31:15.124880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.124845    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main" containerID="cri-o://50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d" gracePeriod=45
Nov 01 22:31:15.427649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.427618    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:31:15.430956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.430936    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:31:15.538083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.537999    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538122    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538206    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538264 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538237    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538264    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.538469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538313    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.538469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538387    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\" (UniqueName: \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.538469 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538435    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\" (UniqueName: \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538521    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\") pod \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\" (UID: \"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6\") "
Nov 01 22:31:15.538682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538624    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.538786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538728    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.538842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.538780    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\") pod \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\" (UID: \"c092b7ee-3175-4c71-8df8-d69b6a6f25d3\") "
Nov 01 22:31:15.540658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.540095    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.540658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.540345    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.540658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.540592    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.541265 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.541175    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.541392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.541325    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.541793 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.541755    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.543521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.543495    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:15.543651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.543624    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "kube-api-access-vthxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:15.544551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.544521    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "kube-api-access-8x8cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:15.544812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.544775    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:15.545513 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.545492    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm" pod "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" (UID: "d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.545598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.545516    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm" pod "c092b7ee-3175-4c71-8df8-d69b6a6f25d3" (UID: "c092b7ee-3175-4c71-8df8-d69b6a6f25d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641283    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641337    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641351    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641364    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641379    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641392    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641405    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641418    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641431    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641455    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x8cm\" (UniqueName: \"kubernetes.io/projected/c092b7ee-3175-4c71-8df8-d69b6a6f25d3-kube-api-access-8x8cm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641469    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthxb\" (UniqueName: \"kubernetes.io/projected/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-kube-api-access-vthxb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.641509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.641482    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:15.653609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.653580    5845 generic.go:356] "Generic (PLEG): container finished" podID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerID="2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08" exitCode=0
Nov 01 22:31:15.653737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.653627    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"
Nov 01 22:31:15.653737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.653661    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" event={"ID":"c092b7ee-3175-4c71-8df8-d69b6a6f25d3","Type":"ContainerDied","Data":"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"}
Nov 01 22:31:15.653737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.653701    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr" event={"ID":"c092b7ee-3175-4c71-8df8-d69b6a6f25d3","Type":"ContainerDied","Data":"a745d73141f5f756ef329a13b500106e44071bce89e0743fbfdf8eeec6a5c318"}
Nov 01 22:31:15.653737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.653719    5845 scope.go:122] "RemoveContainer" containerID="2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"
Nov 01 22:31:15.655236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.655213    5845 generic.go:356] "Generic (PLEG): container finished" podID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerID="50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d" exitCode=0
Nov 01 22:31:15.655402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.655252    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"
Nov 01 22:31:15.655402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.655256    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" event={"ID":"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6","Type":"ContainerDied","Data":"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"}
Nov 01 22:31:15.655402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.655282    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs" event={"ID":"d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6","Type":"ContainerDied","Data":"7d2fbd9cc0669c61d1b0a7e9c464ab4cc2c3f5f41c14b9761dd75d17501d1077"}
Nov 01 22:31:15.665423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.665404    5845 scope.go:122] "RemoveContainer" containerID="2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"
Nov 01 22:31:15.665922 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:15.665723    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08\": container with ID starting with 2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08 not found: ID does not exist" containerID="2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"
Nov 01 22:31:15.665922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.665758    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08"} err="failed to get container status \"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08\": rpc error: code = NotFound desc = could not find container \"2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08\": container with ID starting with 2214073c38e569a4eb98b7e227fc6d224d3c52a4abf9dc0f4165c322d2fa6e08 not found: ID does not exist"
Nov 01 22:31:15.665922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.665781    5845 scope.go:122] "RemoveContainer" containerID="50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"
Nov 01 22:31:15.676432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.676408    5845 scope.go:122] "RemoveContainer" containerID="50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"
Nov 01 22:31:15.676779 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:15.676756    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d\": container with ID starting with 50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d not found: ID does not exist" containerID="50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"
Nov 01 22:31:15.676856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.676786    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d"} err="failed to get container status \"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d\": rpc error: code = NotFound desc = could not find container \"50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d\": container with ID starting with 50d9a8a657d8cd8e04230072af96338043ac56ce00038a6a82ce0ca275cca51d not found: ID does not exist"
Nov 01 22:31:15.681573 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.681548    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 22:31:15.685824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.685797    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/simulator-w5-b0b7a3-kserve-744646869f-7shvs"]
Nov 01 22:31:15.696670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.696648    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 22:31:15.701198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:15.701176    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b0b7a3/distinct-w5-b0b7a3-kserve-9956867c-czxbr"]
Nov 01 22:31:16.861218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:16.861187    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" path="/var/lib/kubelet/pods/c092b7ee-3175-4c71-8df8-d69b6a6f25d3/volumes"
Nov 01 22:31:16.861626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:16.861610    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" path="/var/lib/kubelet/pods/d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6/volumes"
Nov 01 22:31:17.303310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:17.303261    5845 scope.go:122] "RemoveContainer" containerID="1aa19a93aea78a20c14527807ea35d805bf9e150d870bfb3cf6cbc1e53d0609f"
Nov 01 22:31:17.680633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:17.680546    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:31:17.680867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:17.680814    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" podUID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" containerName="payload-processing" containerID="cri-o://a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738" gracePeriod=30
Nov 01 22:31:18.048549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.048517    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:31:18.186235 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.186201    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\" (UniqueName: \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\") pod \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") "
Nov 01 22:31:18.186436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.186392    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\") pod \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\" (UID: \"07d36006-6cbe-4f59-aac1-5b1302e5beb2\") "
Nov 01 22:31:18.186779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.186754    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume" pod "07d36006-6cbe-4f59-aac1-5b1302e5beb2" (UID: "07d36006-6cbe-4f59-aac1-5b1302e5beb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:18.188466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.188441    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm" pod "07d36006-6cbe-4f59-aac1-5b1302e5beb2" (UID: "07d36006-6cbe-4f59-aac1-5b1302e5beb2"). InnerVolumeSpecName "kube-api-access-f92xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:18.287677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.287576    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f92xm\" (UniqueName: \"kubernetes.io/projected/07d36006-6cbe-4f59-aac1-5b1302e5beb2-kube-api-access-f92xm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:18.287677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.287611    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07d36006-6cbe-4f59-aac1-5b1302e5beb2-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:18.687803 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.687707    5845 generic.go:356] "Generic (PLEG): container finished" podID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" containerID="a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738" exitCode=0
Nov 01 22:31:18.687803 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.687755    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"
Nov 01 22:31:18.688009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.687808    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" event={"ID":"07d36006-6cbe-4f59-aac1-5b1302e5beb2","Type":"ContainerDied","Data":"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"}
Nov 01 22:31:18.688009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.687890    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n" event={"ID":"07d36006-6cbe-4f59-aac1-5b1302e5beb2","Type":"ContainerDied","Data":"04988f6b0518bdfc73e9aacef0f90d26b5b548440b33771441d0a04cce2de2a0"}
Nov 01 22:31:18.688009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.687916    5845 scope.go:122] "RemoveContainer" containerID="a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"
Nov 01 22:31:18.698590 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.698570    5845 scope.go:122] "RemoveContainer" containerID="a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"
Nov 01 22:31:18.698863 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:18.698841    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738\": container with ID starting with a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738 not found: ID does not exist" containerID="a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"
Nov 01 22:31:18.698930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.698872    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738"} err="failed to get container status \"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738\": rpc error: code = NotFound desc = could not find container \"a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738\": container with ID starting with a09664c76767d3a7e23706fe96ba8db0e1360de53c179a1d172efc2ad391b738 not found: ID does not exist"
Nov 01 22:31:18.711614 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.711590    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:31:18.715549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.715529    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-09656e-7bd85bc696-r5c8n"]
Nov 01 22:31:18.861672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:18.861642    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" path="/var/lib/kubelet/pods/07d36006-6cbe-4f59-aac1-5b1302e5beb2/volumes"
Nov 01 22:31:32.031916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.031879    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:31:32.038166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.032084    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api" containerID="cri-o://c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16" gracePeriod=30
Nov 01 22:31:32.038166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.037292    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:31:32.383240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.383208    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:31:32.520271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.520239    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\") pod \"9240b129-f08e-44df-9f42-da8c7cae75ef\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") "
Nov 01 22:31:32.520466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.520281    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\") pod \"9240b129-f08e-44df-9f42-da8c7cae75ef\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") "
Nov 01 22:31:32.520466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.520438    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\") pod \"9240b129-f08e-44df-9f42-da8c7cae75ef\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") "
Nov 01 22:31:32.520565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.520513    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\" (UniqueName: \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\") pod \"9240b129-f08e-44df-9f42-da8c7cae75ef\" (UID: \"9240b129-f08e-44df-9f42-da8c7cae75ef\") "
Nov 01 22:31:32.520712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.520684    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca" pod "9240b129-f08e-44df-9f42-da8c7cae75ef" (UID: "9240b129-f08e-44df-9f42-da8c7cae75ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:32.521035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.521009    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9240b129-f08e-44df-9f42-da8c7cae75ef-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:32.522504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.522482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs" pod "9240b129-f08e-44df-9f42-da8c7cae75ef" (UID: "9240b129-f08e-44df-9f42-da8c7cae75ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:32.522611 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.522525    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls" pod "9240b129-f08e-44df-9f42-da8c7cae75ef" (UID: "9240b129-f08e-44df-9f42-da8c7cae75ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:32.522675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.522618    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb" pod "9240b129-f08e-44df-9f42-da8c7cae75ef" (UID: "9240b129-f08e-44df-9f42-da8c7cae75ef"). InnerVolumeSpecName "kube-api-access-v79cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:32.622842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.622763    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:32.622842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.622792    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9240b129-f08e-44df-9f42-da8c7cae75ef-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:32.622842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.622803    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79cb\" (UniqueName: \"kubernetes.io/projected/9240b129-f08e-44df-9f42-da8c7cae75ef-kube-api-access-v79cb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:32.812418 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.812379    5845 generic.go:356] "Generic (PLEG): container finished" podID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerID="c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16" exitCode=0
Nov 01 22:31:32.812592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.812428    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"
Nov 01 22:31:32.812592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.812475    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" event={"ID":"9240b129-f08e-44df-9f42-da8c7cae75ef","Type":"ContainerDied","Data":"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"}
Nov 01 22:31:32.812592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.812528    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh" event={"ID":"9240b129-f08e-44df-9f42-da8c7cae75ef","Type":"ContainerDied","Data":"93ba0a615562f180aeb3f648ea6252c91a69522a76a44baea36504736bb3c510"}
Nov 01 22:31:32.812592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.812551    5845 scope.go:122] "RemoveContainer" containerID="c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"
Nov 01 22:31:32.825178 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.825150    5845 scope.go:122] "RemoveContainer" containerID="c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"
Nov 01 22:31:32.825510 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:32.825488    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16\": container with ID starting with c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16 not found: ID does not exist" containerID="c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"
Nov 01 22:31:32.825601 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.825517    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16"} err="failed to get container status \"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16\": rpc error: code = NotFound desc = could not find container \"c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16\": container with ID starting with c08824ff90ca41cd6dbc42d5f14a666a701dbdbc18bb865a4de00216fa478c16 not found: ID does not exist"
Nov 01 22:31:32.836624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.836601    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:31:32.840709 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.840686    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-1783e3-595cc7869c-sz4bh"]
Nov 01 22:31:32.861491 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:32.861467    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" path="/var/lib/kubelet/pods/9240b129-f08e-44df-9f42-da8c7cae75ef/volumes"
Nov 01 22:31:33.671352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.671311    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:31:33.671863 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.671593    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy" containerID="cri-o://d3a51cd59a2bc2c8829a376eb955dc8d1cecc88689840331cdaf0858bd311700" gracePeriod=30
Nov 01 22:31:33.681534 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.681508    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:31:33.681741 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.681715    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy" containerID="cri-o://75d7b544b7ed05be4a214de4587ba8a0bf12eda1b04faa8320f26d8b8b8821de" gracePeriod=30
Nov 01 22:31:33.688623 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.688597    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:31:33.688829 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:33.688808    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy" containerID="cri-o://3846430b5c367ad4fae42684a8cc92b6e7441b3949aacd42e753d4b4d6095097" gracePeriod=30
Nov 01 22:31:34.816500 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:34.816457    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:31:37.038323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:37.038233    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" containerID="cri-o://4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec" gracePeriod=25
Nov 01 22:31:38.865042 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.865006    5845 generic.go:356] "Generic (PLEG): container finished" podID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerID="3846430b5c367ad4fae42684a8cc92b6e7441b3949aacd42e753d4b4d6095097" exitCode=0
Nov 01 22:31:38.865486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.865337    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerDied","Data":"3846430b5c367ad4fae42684a8cc92b6e7441b3949aacd42e753d4b4d6095097"}
Nov 01 22:31:38.865486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.865389    5845 scope.go:122] "RemoveContainer" containerID="6efd6b77be58641f83b2c999b9d033877be00225997d0bedafb510ce54e4f838"
Nov 01 22:31:38.867210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.867153    5845 generic.go:356] "Generic (PLEG): container finished" podID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerID="75d7b544b7ed05be4a214de4587ba8a0bf12eda1b04faa8320f26d8b8b8821de" exitCode=0
Nov 01 22:31:38.867380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.867232    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" event={"ID":"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8","Type":"ContainerDied","Data":"75d7b544b7ed05be4a214de4587ba8a0bf12eda1b04faa8320f26d8b8b8821de"}
Nov 01 22:31:38.869452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.869414    5845 generic.go:356] "Generic (PLEG): container finished" podID="f934630d-e314-4d71-90f1-539f20f97241" containerID="d3a51cd59a2bc2c8829a376eb955dc8d1cecc88689840331cdaf0858bd311700" exitCode=0
Nov 01 22:31:38.869452 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.869447    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" event={"ID":"f934630d-e314-4d71-90f1-539f20f97241","Type":"ContainerDied","Data":"d3a51cd59a2bc2c8829a376eb955dc8d1cecc88689840331cdaf0858bd311700"}
Nov 01 22:31:38.989376 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.989349    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:31:38.999725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.999701    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:38.999921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.999793    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:38.999921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.999843    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:38.999921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.999875    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:38.999971    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000020    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000054    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000187    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000336 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000253    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\") pod \"f934630d-e314-4d71-90f1-539f20f97241\" (UID: \"f934630d-e314-4d71-90f1-539f20f97241\") "
Nov 01 22:31:39.000804 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000647    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:39.000905 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000823    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f934630d-e314-4d71-90f1-539f20f97241-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.000973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.000951    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.001162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.001144    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.001675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.001422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.001675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.001451    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.004058 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.004023    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "kube-api-access-2x88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.004162 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.004105    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:39.004779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.004741    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.005086 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.004987    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token" pod "f934630d-e314-4d71-90f1-539f20f97241" (UID: "f934630d-e314-4d71-90f1-539f20f97241"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.037628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.037604    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:31:39.049975 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.049946    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:31:39.101486 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101455    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101495    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101515    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101562    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.101679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101603    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.101679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101633    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101692    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101736    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101768    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101788    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101827    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101912    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101931    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.101958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101949    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.102458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.101977    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.102458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102011    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\") pod \"6625bced-23e1-4099-9b61-625eb9f9e05f\" (UID: \"6625bced-23e1-4099-9b61-625eb9f9e05f\") "
Nov 01 22:31:39.102458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102037    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.102458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102065    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\") pod \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\" (UID: \"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8\") "
Nov 01 22:31:39.102458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102372    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102461    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102662    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102674    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102698    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102715    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102728    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102743    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102757    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102770    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102784    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f934630d-e314-4d71-90f1-539f20f97241-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102801    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f934630d-e314-4d71-90f1-539f20f97241-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.102814    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x88x\" (UniqueName: \"kubernetes.io/projected/f934630d-e314-4d71-90f1-539f20f97241-kube-api-access-2x88x\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103015    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103186    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.103282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103196    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.104230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103568    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:39.104230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103626    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.104230 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.103897    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.104962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.104936    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:39.106397 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.106353    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.106537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.106391    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:39.106537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.106483    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.106739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.106612    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "kube-api-access-9ml88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.107201 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.107177    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy" pod "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" (UID: "8af9a875-0b45-4e56-bb74-9ea2ff7f53a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:39.107538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.107513    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "kube-api-access-nsj77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.107631 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.107518    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:39.107911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.107878    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token" pod "6625bced-23e1-4099-9b61-625eb9f9e05f" (UID: "6625bced-23e1-4099-9b61-625eb9f9e05f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204472    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204507    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204526    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204539    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204552    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204566    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204579    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204592    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204605    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204618    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204632    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625bced-23e1-4099-9b61-625eb9f9e05f-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204645    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625bced-23e1-4099-9b61-625eb9f9e05f-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204658    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204672    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204686    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ml88\" (UniqueName: \"kubernetes.io/projected/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8-kube-api-access-9ml88\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.204773 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.204700    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj77\" (UniqueName: \"kubernetes.io/projected/6625bced-23e1-4099-9b61-625eb9f9e05f-kube-api-access-nsj77\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:39.877286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.877254    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"
Nov 01 22:31:39.877745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.877291    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq" event={"ID":"6625bced-23e1-4099-9b61-625eb9f9e05f","Type":"ContainerDied","Data":"a87a6440b71e063bd06a461dc2868d78d938c864a8af2aac702dcd9031719190"}
Nov 01 22:31:39.877745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.877366    5845 scope.go:122] "RemoveContainer" containerID="3846430b5c367ad4fae42684a8cc92b6e7441b3949aacd42e753d4b4d6095097"
Nov 01 22:31:39.878938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.878796    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"
Nov 01 22:31:39.878938 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.878796    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd" event={"ID":"8af9a875-0b45-4e56-bb74-9ea2ff7f53a8","Type":"ContainerDied","Data":"e0085e012439336f70c0680e1891cdf07fb68ef35d84f8a4f213e26a0928da4d"}
Nov 01 22:31:39.880405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.880380    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27" event={"ID":"f934630d-e314-4d71-90f1-539f20f97241","Type":"ContainerDied","Data":"54b66b2364098a16c3a68fd326cd65577f19b62e71c9f721f0d76759b032f0fd"}
Nov 01 22:31:39.880545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.880420    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"
Nov 01 22:31:39.890472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.890384    5845 scope.go:122] "RemoveContainer" containerID="75d7b544b7ed05be4a214de4587ba8a0bf12eda1b04faa8320f26d8b8b8821de"
Nov 01 22:31:39.899889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.899872    5845 scope.go:122] "RemoveContainer" containerID="d3a51cd59a2bc2c8829a376eb955dc8d1cecc88689840331cdaf0858bd311700"
Nov 01 22:31:39.906411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.906390    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:31:39.918156 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.918136    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-5hvsq"]
Nov 01 22:31:39.929528 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.929505    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:31:39.931619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.931600    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-wwm27"]
Nov 01 22:31:39.944686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.944667    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:31:39.959800 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:39.959753    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-521734-openshift-default-756d487fd9-l4fsd"]
Nov 01 22:31:40.860856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:40.860825    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" path="/var/lib/kubelet/pods/6625bced-23e1-4099-9b61-625eb9f9e05f/volumes"
Nov 01 22:31:40.861353 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:40.861337    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" path="/var/lib/kubelet/pods/8af9a875-0b45-4e56-bb74-9ea2ff7f53a8/volumes"
Nov 01 22:31:40.861769 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:40.861757    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f934630d-e314-4d71-90f1-539f20f97241" path="/var/lib/kubelet/pods/f934630d-e314-4d71-90f1-539f20f97241/volumes"
Nov 01 22:31:41.009338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.001770    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:31:41.009338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.002724    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy" containerID="cri-o://b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3" gracePeriod=30
Nov 01 22:31:41.022332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.022104    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:31:41.022473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.022443    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy" containerID="cri-o://fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c" gracePeriod=30
Nov 01 22:31:41.024772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.024749    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:31:41.024961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:41.024940    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy" containerID="cri-o://7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46" gracePeriod=30
Nov 01 22:31:44.293821 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:44.293786    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:31:44.310197 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:44.310165    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:31:45.317423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.317282    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:31:45.317921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.317609    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api" containerID="cri-o://95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69" gracePeriod=30
Nov 01 22:31:45.608020 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.607992    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:31:45.681636 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.681607    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\") pod \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") "
Nov 01 22:31:45.681798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.681648    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\") pod \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") "
Nov 01 22:31:45.681798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.681692    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\") pod \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") "
Nov 01 22:31:45.681798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.681789    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\" (UniqueName: \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\") pod \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\" (UID: \"dd3e2e23-6f3d-48cb-9e35-574acedff5e0\") "
Nov 01 22:31:45.682107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.682074    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca" pod "dd3e2e23-6f3d-48cb-9e35-574acedff5e0" (UID: "dd3e2e23-6f3d-48cb-9e35-574acedff5e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:45.682384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.682357    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:45.683827 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.683800    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls" pod "dd3e2e23-6f3d-48cb-9e35-574acedff5e0" (UID: "dd3e2e23-6f3d-48cb-9e35-574acedff5e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:45.683929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.683819    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs" pod "dd3e2e23-6f3d-48cb-9e35-574acedff5e0" (UID: "dd3e2e23-6f3d-48cb-9e35-574acedff5e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:45.683929 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.683891    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb" pod "dd3e2e23-6f3d-48cb-9e35-574acedff5e0" (UID: "dd3e2e23-6f3d-48cb-9e35-574acedff5e0"). InnerVolumeSpecName "kube-api-access-ldsxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:45.784008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.783967    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldsxb\" (UniqueName: \"kubernetes.io/projected/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-kube-api-access-ldsxb\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:45.784008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.784005    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:45.784008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.784016    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3e2e23-6f3d-48cb-9e35-574acedff5e0-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:45.920792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.920706    5845 generic.go:356] "Generic (PLEG): container finished" podID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerID="95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69" exitCode=0
Nov 01 22:31:45.920792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.920756    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"
Nov 01 22:31:45.920792 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.920779    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" event={"ID":"dd3e2e23-6f3d-48cb-9e35-574acedff5e0","Type":"ContainerDied","Data":"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"}
Nov 01 22:31:45.921034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.920830    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6" event={"ID":"dd3e2e23-6f3d-48cb-9e35-574acedff5e0","Type":"ContainerDied","Data":"aa7d1e1c34df956a2089d2e339cba53bcecf31142d4fd10430da709c50d849d5"}
Nov 01 22:31:45.921034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.920862    5845 scope.go:122] "RemoveContainer" containerID="95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"
Nov 01 22:31:45.931643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.931616    5845 scope.go:122] "RemoveContainer" containerID="95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"
Nov 01 22:31:45.931923 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:45.931902    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69\": container with ID starting with 95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69 not found: ID does not exist" containerID="95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"
Nov 01 22:31:45.931993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.931935    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69"} err="failed to get container status \"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69\": rpc error: code = NotFound desc = could not find container \"95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69\": container with ID starting with 95507d292d1305550d5a0683ed477f26a16329ffe2c30df522d2ec557cbc5e69 not found: ID does not exist"
Nov 01 22:31:45.944112 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.944087    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:31:45.952592 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.952570    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d2832c-58db6b97db-7t6r6"]
Nov 01 22:31:45.952981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:45.952954    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:46.296323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.291688    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:31:46.392703 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.392659    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393239 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393154    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393355 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393280    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393353    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393399    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393429    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393466    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393547    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.393637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.393609    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\") pod \"3574beae-8658-4d85-9d12-255ade49f917\" (UID: \"3574beae-8658-4d85-9d12-255ade49f917\") "
Nov 01 22:31:46.395523 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.394543    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.395620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.395224    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.395620 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.395273    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.396015 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.395990    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.396287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.396264    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:46.396557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.396539    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "kube-api-access-6zch2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.396913 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.396887    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.397069 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.397054    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.397955 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.397937    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo" pod "3574beae-8658-4d85-9d12-255ade49f917" (UID: "3574beae-8658-4d85-9d12-255ade49f917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:46.415033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.415011    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:31:46.418007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.417989    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:31:46.494944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.494909    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.494959    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495018    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495068    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495110 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495104    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495160    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495186    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495243    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495280    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495346    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495409    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495491    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495494    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495543    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495596    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.495628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495621    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495666    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495709    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.495959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495745    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\") pod \"426bc111-0f0a-4c63-a50a-519e47476539\" (UID: \"426bc111-0f0a-4c63-a50a-519e47476539\") "
Nov 01 22:31:46.495959 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.495772    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\") pod \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\" (UID: \"cf9a1de9-e72f-488f-8c08-83a4de512f2e\") "
Nov 01 22:31:46.496341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496319    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496346    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496362    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3574beae-8658-4d85-9d12-255ade49f917-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496372    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496381    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3574beae-8658-4d85-9d12-255ade49f917-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496394    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496401    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496407    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496451    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zch2\" (UniqueName: \"kubernetes.io/projected/3574beae-8658-4d85-9d12-255ade49f917-kube-api-access-6zch2\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496468    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496484    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496501    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3574beae-8658-4d85-9d12-255ade49f917-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.496535 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.497664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.496684    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.497664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.497195    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.497664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.497314    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.497840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.497731    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.498441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498422    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.498569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498480    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.498636 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498597    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:46.498764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498746    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:46.498817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498763    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:46.498872 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.498812    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.499204 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.499189    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.499626 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.499607    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw" pod "426bc111-0f0a-4c63-a50a-519e47476539" (UID: "426bc111-0f0a-4c63-a50a-519e47476539"). InnerVolumeSpecName "kube-api-access-c6kjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.500013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.499996    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "kube-api-access-wkqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:46.500216 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.500197    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo" pod "cf9a1de9-e72f-488f-8c08-83a4de512f2e" (UID: "cf9a1de9-e72f-488f-8c08-83a4de512f2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:46.598802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598760    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.598802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598793    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqjv\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-kube-api-access-wkqjv\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.598802 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598810    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598824    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598838    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598853    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598867    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598880    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598894    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426bc111-0f0a-4c63-a50a-519e47476539-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598907    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598921    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426bc111-0f0a-4c63-a50a-519e47476539-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598934    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598950    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598963    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9a1de9-e72f-488f-8c08-83a4de512f2e-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598976    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6kjw\" (UniqueName: \"kubernetes.io/projected/426bc111-0f0a-4c63-a50a-519e47476539-kube-api-access-c6kjw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.599067 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.598990    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426bc111-0f0a-4c63-a50a-519e47476539-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:46.861940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.861861    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" path="/var/lib/kubelet/pods/dd3e2e23-6f3d-48cb-9e35-574acedff5e0/volumes"
Nov 01 22:31:46.930410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.930371    5845 generic.go:356] "Generic (PLEG): container finished" podID="426bc111-0f0a-4c63-a50a-519e47476539" containerID="7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46" exitCode=0
Nov 01 22:31:46.930555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.930428    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"
Nov 01 22:31:46.930555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.930467    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerDied","Data":"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"}
Nov 01 22:31:46.930555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.930530    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh" event={"ID":"426bc111-0f0a-4c63-a50a-519e47476539","Type":"ContainerDied","Data":"3709df0c16b83d3f2f7063a970977a97de355cbe4d81a05bd808fd813ef50c51"}
Nov 01 22:31:46.930771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.930559    5845 scope.go:122] "RemoveContainer" containerID="7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"
Nov 01 22:31:46.932784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.932699    5845 generic.go:356] "Generic (PLEG): container finished" podID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerID="b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3" exitCode=0
Nov 01 22:31:46.932952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.932783    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"
Nov 01 22:31:46.932952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.932792    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" event={"ID":"cf9a1de9-e72f-488f-8c08-83a4de512f2e","Type":"ContainerDied","Data":"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"}
Nov 01 22:31:46.932952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.932834    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz" event={"ID":"cf9a1de9-e72f-488f-8c08-83a4de512f2e","Type":"ContainerDied","Data":"4dda4f369f4799601659c6cdfe866625a62fcc0363c7d408ac575e5db5062a30"}
Nov 01 22:31:46.936114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.936001    5845 generic.go:356] "Generic (PLEG): container finished" podID="3574beae-8658-4d85-9d12-255ade49f917" containerID="fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c" exitCode=0
Nov 01 22:31:46.936114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.936024    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" event={"ID":"3574beae-8658-4d85-9d12-255ade49f917","Type":"ContainerDied","Data":"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"}
Nov 01 22:31:46.936114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.936074    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv" event={"ID":"3574beae-8658-4d85-9d12-255ade49f917","Type":"ContainerDied","Data":"4e1e72afb5e2086be7a368f2b13c528995bc0e7a07532faf59631d7174a6fbff"}
Nov 01 22:31:46.942025 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.942003    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"
Nov 01 22:31:46.946772 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.946748    5845 scope.go:122] "RemoveContainer" containerID="569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"
Nov 01 22:31:46.979359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.979326    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:31:46.982259 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.982232    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-f6hxz"]
Nov 01 22:31:46.994407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.994163    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:31:46.996468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.996436    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-kq7dh"]
Nov 01 22:31:46.997423 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.997398    5845 scope.go:122] "RemoveContainer" containerID="7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"
Nov 01 22:31:46.997803 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:46.997783    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46\": container with ID starting with 7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46 not found: ID does not exist" containerID="7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"
Nov 01 22:31:46.997889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.997811    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46"} err="failed to get container status \"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46\": rpc error: code = NotFound desc = could not find container \"7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46\": container with ID starting with 7e74a088043a782cf134a67f8a8ab6330c9f98fa979bbd97b11210f688a6bc46 not found: ID does not exist"
Nov 01 22:31:46.997889 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.997828    5845 scope.go:122] "RemoveContainer" containerID="569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"
Nov 01 22:31:46.998123 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:46.998105    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5\": container with ID starting with 569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5 not found: ID does not exist" containerID="569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"
Nov 01 22:31:46.998179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.998130    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5"} err="failed to get container status \"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5\": rpc error: code = NotFound desc = could not find container \"569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5\": container with ID starting with 569c3819021483addb18d5d7b830bec8e0f0cf593bd8b173c3fff36442a867b5 not found: ID does not exist"
Nov 01 22:31:46.998179 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:46.998146    5845 scope.go:122] "RemoveContainer" containerID="b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"
Nov 01 22:31:47.010306 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.010273    5845 scope.go:122] "RemoveContainer" containerID="b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"
Nov 01 22:31:47.010683 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:47.010660    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3\": container with ID starting with b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3 not found: ID does not exist" containerID="b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"
Nov 01 22:31:47.010819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.010691    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3"} err="failed to get container status \"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3\": rpc error: code = NotFound desc = could not find container \"b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3\": container with ID starting with b3880d0aebcdceb58c33d96890333153c279da2a5a706f1b32efc572f49c65e3 not found: ID does not exist"
Nov 01 22:31:47.010819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.010717    5845 scope.go:122] "RemoveContainer" containerID="fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"
Nov 01 22:31:47.011869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.011847    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:31:47.016544 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.016523    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1783e3-openshift-default-6d6647c4cc-5nfvv"]
Nov 01 22:31:47.024043 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.024011    5845 scope.go:122] "RemoveContainer" containerID="fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"
Nov 01 22:31:47.024361 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:47.024288    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c\": container with ID starting with fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c not found: ID does not exist" containerID="fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"
Nov 01 22:31:47.024447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:47.024375    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c"} err="failed to get container status \"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c\": rpc error: code = NotFound desc = could not find container \"fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c\": container with ID starting with fe82250f85f341567ba8fb4c3d02e809c46314f64bca6d7409aca752918db36c not found: ID does not exist"
Nov 01 22:31:48.860858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:48.860821    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3574beae-8658-4d85-9d12-255ade49f917" path="/var/lib/kubelet/pods/3574beae-8658-4d85-9d12-255ade49f917/volumes"
Nov 01 22:31:48.861928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:48.861910    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426bc111-0f0a-4c63-a50a-519e47476539" path="/var/lib/kubelet/pods/426bc111-0f0a-4c63-a50a-519e47476539/volumes"
Nov 01 22:31:48.862391 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:48.862377    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" path="/var/lib/kubelet/pods/cf9a1de9-e72f-488f-8c08-83a4de512f2e/volumes"
Nov 01 22:31:49.294515 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:49.294474    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing" containerID="cri-o://4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68" gracePeriod=25
Nov 01 22:31:49.311103 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:49.311072    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing" containerID="cri-o://53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3" gracePeriod=25
Nov 01 22:31:51.568651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:51.568617    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:31:51.958231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:51.958131    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:31:51.958441 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:51.958410    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api" containerID="cri-o://fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199" gracePeriod=30
Nov 01 22:31:52.265746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.265677    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:31:52.368391 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.368359    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\" (UniqueName: \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\") pod \"a2c93af5-3488-4d83-960b-ead7d2f593e4\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") "
Nov 01 22:31:52.368391 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.368395    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\") pod \"a2c93af5-3488-4d83-960b-ead7d2f593e4\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") "
Nov 01 22:31:52.368606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.368492    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\") pod \"a2c93af5-3488-4d83-960b-ead7d2f593e4\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") "
Nov 01 22:31:52.368606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.368579    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\") pod \"a2c93af5-3488-4d83-960b-ead7d2f593e4\" (UID: \"a2c93af5-3488-4d83-960b-ead7d2f593e4\") "
Nov 01 22:31:52.368810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.368784    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca" pod "a2c93af5-3488-4d83-960b-ead7d2f593e4" (UID: "a2c93af5-3488-4d83-960b-ead7d2f593e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:52.369083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.369063    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c93af5-3488-4d83-960b-ead7d2f593e4-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:52.370656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.370633    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx" pod "a2c93af5-3488-4d83-960b-ead7d2f593e4" (UID: "a2c93af5-3488-4d83-960b-ead7d2f593e4"). InnerVolumeSpecName "kube-api-access-w6lbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:52.370762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.370666    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs" pod "a2c93af5-3488-4d83-960b-ead7d2f593e4" (UID: "a2c93af5-3488-4d83-960b-ead7d2f593e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:52.370762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.370694    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls" pod "a2c93af5-3488-4d83-960b-ead7d2f593e4" (UID: "a2c93af5-3488-4d83-960b-ead7d2f593e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:52.470388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.470353    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6lbx\" (UniqueName: \"kubernetes.io/projected/a2c93af5-3488-4d83-960b-ead7d2f593e4-kube-api-access-w6lbx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:52.470388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.470380    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:52.470388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.470389    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c93af5-3488-4d83-960b-ead7d2f593e4-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:52.982873 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.982836    5845 generic.go:356] "Generic (PLEG): container finished" podID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerID="fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199" exitCode=0
Nov 01 22:31:52.983341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.982882    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"
Nov 01 22:31:52.983341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.982921    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" event={"ID":"a2c93af5-3488-4d83-960b-ead7d2f593e4","Type":"ContainerDied","Data":"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"}
Nov 01 22:31:52.983341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.982981    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr" event={"ID":"a2c93af5-3488-4d83-960b-ead7d2f593e4","Type":"ContainerDied","Data":"7892d597b0675245468aa9ae44e1e0fed63dca26333971b5c6e915631858271a"}
Nov 01 22:31:52.983341 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.983011    5845 scope.go:122] "RemoveContainer" containerID="fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"
Nov 01 22:31:52.993930 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.993911    5845 scope.go:122] "RemoveContainer" containerID="fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"
Nov 01 22:31:52.994211 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:52.994192    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199\": container with ID starting with fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199 not found: ID does not exist" containerID="fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"
Nov 01 22:31:52.994267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:52.994221    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199"} err="failed to get container status \"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199\": rpc error: code = NotFound desc = could not find container \"fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199\": container with ID starting with fdebf15b348e561908b85d5b257a11735f754112564e046feb226228e374b199 not found: ID does not exist"
Nov 01 22:31:53.004902 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:53.004874    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:31:53.009891 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:53.009868    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-e2bf63-6b857d848f-ljptr"]
Nov 01 22:31:53.318345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:53.318314    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:31:53.318545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:53.318525    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" containerID="cri-o://d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69" gracePeriod=30
Nov 01 22:31:54.860745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:54.860712    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" path="/var/lib/kubelet/pods/a2c93af5-3488-4d83-960b-ead7d2f593e4/volumes"
Nov 01 22:31:55.686477 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:55.686440    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:55.686477 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:55.686465    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:55.953458 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:55.953376    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:31:56.569748 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:56.569707    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing" containerID="cri-o://76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758" gracePeriod=25
Nov 01 22:31:57.129712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.129671    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:31:57.130077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.129735    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:31:57.297191 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.296815    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:31:57.427026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.426998    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\") pod \"62b6f384-447a-44d8-b2a0-dc547696f39b\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") "
Nov 01 22:31:57.427171 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.427128    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\" (UniqueName: \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\") pod \"62b6f384-447a-44d8-b2a0-dc547696f39b\" (UID: \"62b6f384-447a-44d8-b2a0-dc547696f39b\") "
Nov 01 22:31:57.427386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.427363    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume" pod "62b6f384-447a-44d8-b2a0-dc547696f39b" (UID: "62b6f384-447a-44d8-b2a0-dc547696f39b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:57.429794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.429764    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667" pod "62b6f384-447a-44d8-b2a0-dc547696f39b" (UID: "62b6f384-447a-44d8-b2a0-dc547696f39b"). InnerVolumeSpecName "kube-api-access-gq667". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:57.528627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.528585    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62b6f384-447a-44d8-b2a0-dc547696f39b-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:57.528627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:57.528622    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq667\" (UniqueName: \"kubernetes.io/projected/62b6f384-447a-44d8-b2a0-dc547696f39b-kube-api-access-gq667\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.014640 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.014605    5845 generic.go:356] "Generic (PLEG): container finished" podID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerID="4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec" exitCode=0
Nov 01 22:31:58.014834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.014657    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"
Nov 01 22:31:58.014834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.014689    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" event={"ID":"62b6f384-447a-44d8-b2a0-dc547696f39b","Type":"ContainerDied","Data":"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"}
Nov 01 22:31:58.014834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.014749    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2" event={"ID":"62b6f384-447a-44d8-b2a0-dc547696f39b","Type":"ContainerDied","Data":"554e2fabb6f6e2d966e6aa379f5b520b0fc80a00e100ddf6fa407aeea5be2a08"}
Nov 01 22:31:58.014834 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.014779    5845 scope.go:122] "RemoveContainer" containerID="4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"
Nov 01 22:31:58.025357 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.025334    5845 scope.go:122] "RemoveContainer" containerID="4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"
Nov 01 22:31:58.025640 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:58.025621    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec\": container with ID starting with 4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec not found: ID does not exist" containerID="4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"
Nov 01 22:31:58.025706 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.025650    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec"} err="failed to get container status \"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec\": rpc error: code = NotFound desc = could not find container \"4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec\": container with ID starting with 4a1d4062e909abbab519dd2f0c0eef9da1444f6f247d0cf53976e350e61ee4ec not found: ID does not exist"
Nov 01 22:31:58.036705 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.036684    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:31:58.040804 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.040785    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-1783e3-6b45cff6f9-rbgv2"]
Nov 01 22:31:58.610157 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.610095    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.767724    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.767816    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.767900    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.767942    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.767991    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.768021    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.768049    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.768101    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.789354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.768168    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\") pod \"7366d1dd-7a69-47c6-983f-fb72ddccce54\" (UID: \"7366d1dd-7a69-47c6-983f-fb72ddccce54\") "
Nov 01 22:31:58.801647 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.801388    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:58.802188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.801871    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:58.802764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.802482    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:58.809104 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.802895    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:58.823472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.815348    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:58.849497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.837730    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:31:58.849497 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.840418    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.885801    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886783    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886808    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886823    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886836    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886850    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886864    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7366d1dd-7a69-47c6-983f-fb72ddccce54-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886878    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.897362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.886892    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7366d1dd-7a69-47c6-983f-fb72ddccce54-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:58.925591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.904797    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6" pod "7366d1dd-7a69-47c6-983f-fb72ddccce54" (UID: "7366d1dd-7a69-47c6-983f-fb72ddccce54"). InnerVolumeSpecName "kube-api-access-5qdm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:58.925591 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.917922    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" path="/var/lib/kubelet/pods/62b6f384-447a-44d8-b2a0-dc547696f39b/volumes"
Nov 01 22:31:58.988145 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:58.988104    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qdm6\" (UniqueName: \"kubernetes.io/projected/7366d1dd-7a69-47c6-983f-fb72ddccce54-kube-api-access-5qdm6\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:59.022920 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.022889    5845 generic.go:356] "Generic (PLEG): container finished" podID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerID="d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69" exitCode=0
Nov 01 22:31:59.023085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.022940    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"
Nov 01 22:31:59.023085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.022979    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerDied","Data":"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"}
Nov 01 22:31:59.023085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.023038    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb" event={"ID":"7366d1dd-7a69-47c6-983f-fb72ddccce54","Type":"ContainerDied","Data":"768d80c42efe7f6a3d9d84b64a865731b1c30f4e73e72f4844201c7ccaa420e4"}
Nov 01 22:31:59.023085 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.023056    5845 scope.go:122] "RemoveContainer" containerID="d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"
Nov 01 22:31:59.034327 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.034306    5845 scope.go:122] "RemoveContainer" containerID="f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"
Nov 01 22:31:59.041688 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.041663    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:31:59.046062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.046044    5845 scope.go:122] "RemoveContainer" containerID="d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"
Nov 01 22:31:59.046397 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:59.046365    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69\": container with ID starting with d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69 not found: ID does not exist" containerID="d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"
Nov 01 22:31:59.048369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.046403    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69"} err="failed to get container status \"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69\": rpc error: code = NotFound desc = could not find container \"d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69\": container with ID starting with d40dc0751971d924c9dd14e8c115e81e7dc14ba78d3abbe94bc31ed075bfdf69 not found: ID does not exist"
Nov 01 22:31:59.048369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.047207    5845 scope.go:122] "RemoveContainer" containerID="f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"
Nov 01 22:31:59.048369 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:31:59.047593    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826\": container with ID starting with f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826 not found: ID does not exist" containerID="f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"
Nov 01 22:31:59.048369 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.047616    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826"} err="failed to get container status \"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826\": rpc error: code = NotFound desc = could not find container \"f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826\": container with ID starting with f8c068ec7047df47f950c4829bb345f4da4a5c117bbb936481a12e1141b05826 not found: ID does not exist"
Nov 01 22:31:59.050144 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:31:59.050119    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d2832c-openshift-default-6d8cc8f567-bj6sb"]
Nov 01 22:32:00.110894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:00.110841    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:32:00.111443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:00.111115    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" containerID="cri-o://b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d" gracePeriod=30
Nov 01 22:32:00.860558 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:00.860528    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" path="/var/lib/kubelet/pods/7366d1dd-7a69-47c6-983f-fb72ddccce54/volumes"
Nov 01 22:32:01.562039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:01.562005    5845 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:32:01.562432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:01.562069    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:32:02.287977 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:02.287942    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:32:04.816439 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:04.816401    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:32:05.378512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.378453    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:32:05.544478 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.544441    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 22:32:05.549504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.549476    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 22:32:05.567647 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567610    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.567838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567666    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.567838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567703    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.567838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567792    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.567838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567827    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.568072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567871    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.568072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567960    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.568072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.567992    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.568072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.568025    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\") pod \"46d4bcd7-f7ed-4f73-95a0-235581345bec\" (UID: \"46d4bcd7-f7ed-4f73-95a0-235581345bec\") "
Nov 01 22:32:05.568072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.568039    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:05.569238 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.569163    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:05.569645 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.569617    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:05.569770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.569739    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:05.570003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.569980    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:05.570629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.570528    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.570629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.570554    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.570629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.570572    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4bcd7-f7ed-4f73-95a0-235581345bec-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.570629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.570586    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.570629 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.570600    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.571652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.571604    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:32:05.572603 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.572559    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:05.572730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.572635    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "kube-api-access-xxzgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:05.574596 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.574579    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy" pod "46d4bcd7-f7ed-4f73-95a0-235581345bec" (UID: "46d4bcd7-f7ed-4f73-95a0-235581345bec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:05.671394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.671320    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.671394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.671345    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.671394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.671355    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4bcd7-f7ed-4f73-95a0-235581345bec-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.671394 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.671367    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzgt\" (UniqueName: \"kubernetes.io/projected/46d4bcd7-f7ed-4f73-95a0-235581345bec-kube-api-access-xxzgt\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:05.687063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.687030    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:32:05.687229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:05.687089    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:32:06.078763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.078722    5845 generic.go:356] "Generic (PLEG): container finished" podID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerID="b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d" exitCode=0
Nov 01 22:32:06.079170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.078780    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"
Nov 01 22:32:06.079170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.078781    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" event={"ID":"46d4bcd7-f7ed-4f73-95a0-235581345bec","Type":"ContainerDied","Data":"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"}
Nov 01 22:32:06.079170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.078950    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv" event={"ID":"46d4bcd7-f7ed-4f73-95a0-235581345bec","Type":"ContainerDied","Data":"ac802febe9d9854a8576456b2a6ee0ef855b08fb1071e389edaaabb7e2cea3d3"}
Nov 01 22:32:06.079170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.078982    5845 scope.go:122] "RemoveContainer" containerID="b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"
Nov 01 22:32:06.098485 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.098460    5845 scope.go:122] "RemoveContainer" containerID="b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"
Nov 01 22:32:06.098933 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:06.098911    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d\": container with ID starting with b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d not found: ID does not exist" containerID="b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"
Nov 01 22:32:06.099073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.098940    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d"} err="failed to get container status \"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d\": rpc error: code = NotFound desc = could not find container \"b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d\": container with ID starting with b0198826a183c9977f9bdb80259514138ce167b84d1d51c1f8986450a1393c6d not found: ID does not exist"
Nov 01 22:32:06.103233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.103208    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:32:06.109352 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.109325    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e2bf63-openshift-default-7cd68f8c89-rlwzv"]
Nov 01 22:32:06.860847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:06.860812    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" path="/var/lib/kubelet/pods/46d4bcd7-f7ed-4f73-95a0-235581345bec/volumes"
Nov 01 22:32:07.850277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:07.850243    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:32:07.850746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:07.850526    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" containerID="cri-o://9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34" gracePeriod=30
Nov 01 22:32:08.344816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:08.344780    5845 patch_prober.go:29] interesting pod/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:32:08.344989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:08.344839    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:32:09.578350 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.578326    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:32:09.612398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.612371    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:32:09.615381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.615358    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\") pod \"9fbe1608-dc84-42dd-83b7-54be275ae505\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") "
Nov 01 22:32:09.615504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.615483    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\" (UniqueName: \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\") pod \"9fbe1608-dc84-42dd-83b7-54be275ae505\" (UID: \"9fbe1608-dc84-42dd-83b7-54be275ae505\") "
Nov 01 22:32:09.615754 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.615730    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume" pod "9fbe1608-dc84-42dd-83b7-54be275ae505" (UID: "9fbe1608-dc84-42dd-83b7-54be275ae505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:09.615875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.615858    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbe1608-dc84-42dd-83b7-54be275ae505-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:09.617557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.617537    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j" pod "9fbe1608-dc84-42dd-83b7-54be275ae505" (UID: "9fbe1608-dc84-42dd-83b7-54be275ae505"). InnerVolumeSpecName "kube-api-access-4sg9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:09.717199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.717163    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\") pod \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") "
Nov 01 22:32:09.717419 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.717337    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\" (UniqueName: \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\") pod \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\" (UID: \"d832d3b8-c3e0-4d1e-a44c-e80aca53a364\") "
Nov 01 22:32:09.717613 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.717585    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume" pod "d832d3b8-c3e0-4d1e-a44c-e80aca53a364" (UID: "d832d3b8-c3e0-4d1e-a44c-e80aca53a364"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:09.717766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.717751    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sg9j\" (UniqueName: \"kubernetes.io/projected/9fbe1608-dc84-42dd-83b7-54be275ae505-kube-api-access-4sg9j\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:09.717806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.717769    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:09.719437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.719414    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm" pod "d832d3b8-c3e0-4d1e-a44c-e80aca53a364" (UID: "d832d3b8-c3e0-4d1e-a44c-e80aca53a364"). InnerVolumeSpecName "kube-api-access-48pkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:09.818420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:09.818362    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pkm\" (UniqueName: \"kubernetes.io/projected/d832d3b8-c3e0-4d1e-a44c-e80aca53a364-kube-api-access-48pkm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:10.103664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.103574    5845 generic.go:356] "Generic (PLEG): container finished" podID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerID="4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68" exitCode=0
Nov 01 22:32:10.103664 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.103626    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"
Nov 01 22:32:10.103882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.103671    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" event={"ID":"9fbe1608-dc84-42dd-83b7-54be275ae505","Type":"ContainerDied","Data":"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"}
Nov 01 22:32:10.103882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.103730    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x" event={"ID":"9fbe1608-dc84-42dd-83b7-54be275ae505","Type":"ContainerDied","Data":"feb2d485c60d8e4d97172f15bfa4dcc12cd7bb2c2effa63357de582437834070"}
Nov 01 22:32:10.103882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.103755    5845 scope.go:122] "RemoveContainer" containerID="4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"
Nov 01 22:32:10.105309 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.105271    5845 generic.go:356] "Generic (PLEG): container finished" podID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerID="53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3" exitCode=0
Nov 01 22:32:10.105440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.105344    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"
Nov 01 22:32:10.105440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.105354    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" event={"ID":"d832d3b8-c3e0-4d1e-a44c-e80aca53a364","Type":"ContainerDied","Data":"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"}
Nov 01 22:32:10.105440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.105414    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c" event={"ID":"d832d3b8-c3e0-4d1e-a44c-e80aca53a364","Type":"ContainerDied","Data":"ac8ec4c1600ad294ff0bc3f7b8ab156a8fdc21ca70fef2b3997fbfe2d02faea6"}
Nov 01 22:32:10.116234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.116214    5845 scope.go:122] "RemoveContainer" containerID="4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"
Nov 01 22:32:10.116575 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:10.116554    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68\": container with ID starting with 4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68 not found: ID does not exist" containerID="4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"
Nov 01 22:32:10.116638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.116585    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68"} err="failed to get container status \"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68\": rpc error: code = NotFound desc = could not find container \"4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68\": container with ID starting with 4de928aa701005e7a65b7ced81a964b067db2946ab9b13e1739db29b0d008e68 not found: ID does not exist"
Nov 01 22:32:10.116638 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.116600    5845 scope.go:122] "RemoveContainer" containerID="53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"
Nov 01 22:32:10.126962 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.126939    5845 scope.go:122] "RemoveContainer" containerID="53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"
Nov 01 22:32:10.127323 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:10.127271    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3\": container with ID starting with 53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3 not found: ID does not exist" containerID="53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"
Nov 01 22:32:10.127402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.127338    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3"} err="failed to get container status \"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3\": rpc error: code = NotFound desc = could not find container \"53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3\": container with ID starting with 53c1a0bd7fe0416ed677b481af884ab92e486345026a2c1cdd4fed26bfd043d3 not found: ID does not exist"
Nov 01 22:32:10.127468 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.127451    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:32:10.132152 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.132132    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d2832c-7584ddcdbd-8qp5x"]
Nov 01 22:32:10.148214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.148194    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:32:10.155713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.155097    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d2832c-c4cc54dc5-m6f9c"]
Nov 01 22:32:10.862646 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.862605    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" path="/var/lib/kubelet/pods/9fbe1608-dc84-42dd-83b7-54be275ae505/volumes"
Nov 01 22:32:10.863217 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:10.863198    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" path="/var/lib/kubelet/pods/d832d3b8-c3e0-4d1e-a44c-e80aca53a364/volumes"
Nov 01 22:32:12.287850 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:12.287812    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:32:13.119006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.118950    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:32:13.127553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.127525    5845 generic.go:356] "Generic (PLEG): container finished" podID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerID="9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34" exitCode=0
Nov 01 22:32:13.127678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.127575    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"
Nov 01 22:32:13.127678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.127606    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerDied","Data":"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"}
Nov 01 22:32:13.127678 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.127667    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm" event={"ID":"b204b3aa-d4fe-48b1-93b7-888db9d93a20","Type":"ContainerDied","Data":"dcf2b19cb28f4fc578470ceae013787462385f34e87eefaad95b0bca47d88fba"}
Nov 01 22:32:13.127814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.127691    5845 scope.go:122] "RemoveContainer" containerID="9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"
Nov 01 22:32:13.137200 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.137181    5845 scope.go:122] "RemoveContainer" containerID="b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"
Nov 01 22:32:13.148403 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.148381    5845 scope.go:122] "RemoveContainer" containerID="9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"
Nov 01 22:32:13.148707 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:13.148679    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34\": container with ID starting with 9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34 not found: ID does not exist" containerID="9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"
Nov 01 22:32:13.148808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.148714    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34"} err="failed to get container status \"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34\": rpc error: code = NotFound desc = could not find container \"9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34\": container with ID starting with 9f78576daf9e32893c1e0b638bc6b8b4830b44e2fceb7b60ae7a4a84e9f6df34 not found: ID does not exist"
Nov 01 22:32:13.148808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.148738    5845 scope.go:122] "RemoveContainer" containerID="b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"
Nov 01 22:32:13.149061 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:13.149041    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1\": container with ID starting with b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1 not found: ID does not exist" containerID="b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"
Nov 01 22:32:13.149119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.149068    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1"} err="failed to get container status \"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1\": rpc error: code = NotFound desc = could not find container \"b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1\": container with ID starting with b9f3dde8c9e610c3589a8d483a2fa73d9af5cf3db79c277ac4e34eb4d41a46e1 not found: ID does not exist"
Nov 01 22:32:13.162605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162577    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.162758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162637    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.162758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162694    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.162758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162740    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.162924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162805    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.162924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.162837    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.163212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163087    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:13.163212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163121    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.163212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163112    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:13.163212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163161    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:13.163212 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163183    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.163649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163215    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\") pod \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\" (UID: \"b204b3aa-d4fe-48b1-93b7-888db9d93a20\") "
Nov 01 22:32:13.163649 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163628    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:13.163771 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163728    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:13.163833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163776    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.163833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163796    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.163833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163812    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.163833 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.163826    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.165124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.165104    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:32:13.165231 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.165166    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:13.165287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.165236    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:13.165676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.165651    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll" pod "b204b3aa-d4fe-48b1-93b7-888db9d93a20" (UID: "b204b3aa-d4fe-48b1-93b7-888db9d93a20"). InnerVolumeSpecName "kube-api-access-x2xll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:13.264602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.264570    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.264602 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.264602    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.264795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.264618    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.264795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.264630    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b204b3aa-d4fe-48b1-93b7-888db9d93a20-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.264795 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.264643    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xll\" (UniqueName: \"kubernetes.io/projected/b204b3aa-d4fe-48b1-93b7-888db9d93a20-kube-api-access-x2xll\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:13.450684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.450648    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:32:13.457140 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:13.457114    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-09656e-openshift-default-6d5c46dfc-t9gnm"]
Nov 01 22:32:14.861240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:14.861196    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" path="/var/lib/kubelet/pods/b204b3aa-d4fe-48b1-93b7-888db9d93a20/volumes"
Nov 01 22:32:16.828504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.828444    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:32:16.908695 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.908670    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\") pod \"2d145ee6-4342-41e4-a95b-9365770eecf7\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") "
Nov 01 22:32:16.908884 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.908710    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\" (UniqueName: \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\") pod \"2d145ee6-4342-41e4-a95b-9365770eecf7\" (UID: \"2d145ee6-4342-41e4-a95b-9365770eecf7\") "
Nov 01 22:32:16.909141 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.909116    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume" pod "2d145ee6-4342-41e4-a95b-9365770eecf7" (UID: "2d145ee6-4342-41e4-a95b-9365770eecf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:16.909286 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.909269    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d145ee6-4342-41e4-a95b-9365770eecf7-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:16.910869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:16.910848    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz" pod "2d145ee6-4342-41e4-a95b-9365770eecf7" (UID: "2d145ee6-4342-41e4-a95b-9365770eecf7"). InnerVolumeSpecName "kube-api-access-zttlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:17.010887 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.010850    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttlz\" (UniqueName: \"kubernetes.io/projected/2d145ee6-4342-41e4-a95b-9365770eecf7-kube-api-access-zttlz\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:17.155985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.155895    5845 generic.go:356] "Generic (PLEG): container finished" podID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerID="76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758" exitCode=0
Nov 01 22:32:17.155985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.155950    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"
Nov 01 22:32:17.156189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.155980    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" event={"ID":"2d145ee6-4342-41e4-a95b-9365770eecf7","Type":"ContainerDied","Data":"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"}
Nov 01 22:32:17.156189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.156069    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2" event={"ID":"2d145ee6-4342-41e4-a95b-9365770eecf7","Type":"ContainerDied","Data":"7770566054bf809af91970722540e8bea8632a4be32c9c9a70ee5da4e4857bf1"}
Nov 01 22:32:17.156189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.156092    5845 scope.go:122] "RemoveContainer" containerID="76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"
Nov 01 22:32:17.166642 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.166626    5845 scope.go:122] "RemoveContainer" containerID="76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"
Nov 01 22:32:17.166898 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:17.166877    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758\": container with ID starting with 76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758 not found: ID does not exist" containerID="76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"
Nov 01 22:32:17.166969 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.166912    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758"} err="failed to get container status \"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758\": rpc error: code = NotFound desc = could not find container \"76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758\": container with ID starting with 76893541842e8486a245f555cee303fe07d59ba04fed15c95458db5fbe43e758 not found: ID does not exist"
Nov 01 22:32:17.179285 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.179261    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:32:17.184154 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:17.184130    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-e2bf63-68fd867c6b-qbtk2"]
Nov 01 22:32:18.859851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:18.859817    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" path="/var/lib/kubelet/pods/2d145ee6-4342-41e4-a95b-9365770eecf7/volumes"
Nov 01 22:32:20.566401 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.566355    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" containerID="cri-o://1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c" gracePeriod=45
Nov 01 22:32:20.572271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.571430    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" containerID="cri-o://35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40" gracePeriod=45
Nov 01 22:32:20.586059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.586025    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused"
Nov 01 22:32:20.588820 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.588795    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused"
Nov 01 22:32:20.851861 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.851832    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:32:20.905186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.902789    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:32:20.952656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952625    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.952656 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952663    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.952918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952753    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.952918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952848    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.952918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952892    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\" (UniqueName: \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.953096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.952930    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\") pod \"d443e8ab-a261-40e1-9884-bbeee643fc4c\" (UID: \"d443e8ab-a261-40e1-9884-bbeee643fc4c\") "
Nov 01 22:32:20.953714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.953683    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:20.953898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.953877    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:20.954318 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.954262    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:20.956529 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.956497    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:20.957662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.957635    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:20.959262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:20.959235    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn" pod "d443e8ab-a261-40e1-9884-bbeee643fc4c" (UID: "d443e8ab-a261-40e1-9884-bbeee643fc4c"). InnerVolumeSpecName "kube-api-access-mgtvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.053915    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054000    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054050    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054080    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054206    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054247    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\" (UniqueName: \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\") pod \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\" (UID: \"3a4cc922-e1aa-4edf-bc11-26809fd8cd10\") "
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054954    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054982    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtvn\" (UniqueName: \"kubernetes.io/projected/d443e8ab-a261-40e1-9884-bbeee643fc4c-kube-api-access-mgtvn\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.054999    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d443e8ab-a261-40e1-9884-bbeee643fc4c-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.055012    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.055025    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.055038    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d443e8ab-a261-40e1-9884-bbeee643fc4c-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.056555    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.056736    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:21.057725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.057008    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:21.064293 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.064262    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:21.064555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.064467    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "kube-api-access-7zghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:21.064555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.064533    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm" pod "3a4cc922-e1aa-4edf-bc11-26809fd8cd10" (UID: "3a4cc922-e1aa-4edf-bc11-26809fd8cd10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155574    5845 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-dshm\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155604    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zghx\" (UniqueName: \"kubernetes.io/projected/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-kube-api-access-7zghx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155619    5845 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-model-cache\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155636    5845 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tls-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155645    5845 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-home\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.155668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.155654    5845 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a4cc922-e1aa-4edf-bc11-26809fd8cd10-tmp-dir\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.182665 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.182623    5845 generic.go:356] "Generic (PLEG): container finished" podID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerID="1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c" exitCode=0
Nov 01 22:32:21.182951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.182679    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"
Nov 01 22:32:21.183107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.182693    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" event={"ID":"d443e8ab-a261-40e1-9884-bbeee643fc4c","Type":"ContainerDied","Data":"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"}
Nov 01 22:32:21.183176 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.183095    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw" event={"ID":"d443e8ab-a261-40e1-9884-bbeee643fc4c","Type":"ContainerDied","Data":"4f465aaa8e1f5e46a8f5f28b616300cf44ce433fc5f4f82299a7b7c6753bfb15"}
Nov 01 22:32:21.183176 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.183144    5845 scope.go:122] "RemoveContainer" containerID="1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"
Nov 01 22:32:21.194172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.190571    5845 generic.go:356] "Generic (PLEG): container finished" podID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerID="35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40" exitCode=0
Nov 01 22:32:21.194172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.190693    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" event={"ID":"3a4cc922-e1aa-4edf-bc11-26809fd8cd10","Type":"ContainerDied","Data":"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"}
Nov 01 22:32:21.194172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.190745    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58" event={"ID":"3a4cc922-e1aa-4edf-bc11-26809fd8cd10","Type":"ContainerDied","Data":"c6aab56c7f0cb593379042ec24e5d149969cc08555157248cff7438a5e369498"}
Nov 01 22:32:21.194172 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.190843    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"
Nov 01 22:32:21.197867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.197844    5845 scope.go:122] "RemoveContainer" containerID="1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"
Nov 01 22:32:21.203058 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:21.203029    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c\": container with ID starting with 1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c not found: ID does not exist" containerID="1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"
Nov 01 22:32:21.203233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.203208    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c"} err="failed to get container status \"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c\": rpc error: code = NotFound desc = could not find container \"1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c\": container with ID starting with 1fc896ea0b0f4da5ea2dbc241ad7523ff0e761b8ac6c5f131546b3d7d6555c2c not found: ID does not exist"
Nov 01 22:32:21.203378 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.203358    5845 scope.go:122] "RemoveContainer" containerID="35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"
Nov 01 22:32:21.216335 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.216291    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 22:32:21.220846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.220814    5845 scope.go:122] "RemoveContainer" containerID="35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"
Nov 01 22:32:21.221208 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:21.221176    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40\": container with ID starting with 35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40 not found: ID does not exist" containerID="35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"
Nov 01 22:32:21.221363 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.221219    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40"} err="failed to get container status \"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40\": rpc error: code = NotFound desc = could not find container \"35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40\": container with ID starting with 35bbfa83117dfe8c17945126a7207c775de5f17492d59aa2f50089c16420cb40 not found: ID does not exist"
Nov 01 22:32:21.222816 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.222793    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-2-w4-0def36-kserve-78448f588-6rxdw"]
Nov 01 22:32:21.249365 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.249337    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 22:32:21.258637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.258605    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0def36/distinct-w4-0def36-kserve-5b8f9dcd99-x9l58"]
Nov 01 22:32:21.487384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.486987    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 22:32:21.487384 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.487267    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api" containerID="cri-o://d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0" gracePeriod=30
Nov 01 22:32:21.771778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.771752    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 22:32:21.868794 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.868762    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\") pod \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") "
Nov 01 22:32:21.869018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.868874    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\") pod \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") "
Nov 01 22:32:21.869018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.868929    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\" (UniqueName: \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\") pod \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") "
Nov 01 22:32:21.869018 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.869001    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\") pod \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\" (UID: \"0e36822e-f396-4ebe-a7b9-0b3a967446c6\") "
Nov 01 22:32:21.871824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.869555    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca" pod "0e36822e-f396-4ebe-a7b9-0b3a967446c6" (UID: "0e36822e-f396-4ebe-a7b9-0b3a967446c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:21.872344 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.872320    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs" pod "0e36822e-f396-4ebe-a7b9-0b3a967446c6" (UID: "0e36822e-f396-4ebe-a7b9-0b3a967446c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:21.873728 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.873679    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw" pod "0e36822e-f396-4ebe-a7b9-0b3a967446c6" (UID: "0e36822e-f396-4ebe-a7b9-0b3a967446c6"). InnerVolumeSpecName "kube-api-access-4hwmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:21.873853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.873735    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls" pod "0e36822e-f396-4ebe-a7b9-0b3a967446c6" (UID: "0e36822e-f396-4ebe-a7b9-0b3a967446c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:21.970958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.970918    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36822e-f396-4ebe-a7b9-0b3a967446c6-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.970958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.970950    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwmw\" (UniqueName: \"kubernetes.io/projected/0e36822e-f396-4ebe-a7b9-0b3a967446c6-kube-api-access-4hwmw\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.970958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.970960    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:21.970958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:21.970969    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36822e-f396-4ebe-a7b9-0b3a967446c6-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:22.201625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.201474    5845 generic.go:356] "Generic (PLEG): container finished" podID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerID="d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0" exitCode=0
Nov 01 22:32:22.201625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.201524    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" event={"ID":"0e36822e-f396-4ebe-a7b9-0b3a967446c6","Type":"ContainerDied","Data":"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"}
Nov 01 22:32:22.201625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.201575    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5" event={"ID":"0e36822e-f396-4ebe-a7b9-0b3a967446c6","Type":"ContainerDied","Data":"ac340f8d33f1d7e83208f03b21b2499511d6c43b166fec7b14667cd612c0a340"}
Nov 01 22:32:22.201625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.201605    5845 scope.go:122] "RemoveContainer" containerID="d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"
Nov 01 22:32:22.201981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.201721    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"
Nov 01 22:32:22.219064 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.219037    5845 scope.go:122] "RemoveContainer" containerID="d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"
Nov 01 22:32:22.219515 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:22.219491    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0\": container with ID starting with d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0 not found: ID does not exist" containerID="d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"
Nov 01 22:32:22.219601 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.219528    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0"} err="failed to get container status \"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0\": rpc error: code = NotFound desc = could not find container \"d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0\": container with ID starting with d5ceec2dd27be42e7d249a72af80b97e4174dba5fdb363649c507fcc6e5a5ae0 not found: ID does not exist"
Nov 01 22:32:22.229684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.229657    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 22:32:22.233634 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.233613    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ff3e70-685b6754b5-vwgq5"]
Nov 01 22:32:22.646472 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.646435    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 22:32:22.647005 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.646970    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api" containerID="cri-o://20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9" gracePeriod=30
Nov 01 22:32:22.840291 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.840037    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:32:22.842165 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.841078    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" podUID="e07fa865-0c48-457d-8dc7-fd430c157021" containerName="payload-processing" containerID="cri-o://0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905" gracePeriod=30
Nov 01 22:32:22.845343 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.845318    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:32:22.845680 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.845652    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" podUID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" containerName="payload-pre-processing" containerID="cri-o://9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c" gracePeriod=30
Nov 01 22:32:22.863619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.863589    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" path="/var/lib/kubelet/pods/0e36822e-f396-4ebe-a7b9-0b3a967446c6/volumes"
Nov 01 22:32:22.864228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.864187    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" path="/var/lib/kubelet/pods/3a4cc922-e1aa-4edf-bc11-26809fd8cd10/volumes"
Nov 01 22:32:22.865453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.864711    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" path="/var/lib/kubelet/pods/d443e8ab-a261-40e1-9884-bbeee643fc4c/volumes"
Nov 01 22:32:22.978434 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:22.978410    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 22:32:23.081220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.081189    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\") pod \"565fc23b-8273-44b8-92c7-eb902f590022\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") "
Nov 01 22:32:23.081417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.081277    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\") pod \"565fc23b-8273-44b8-92c7-eb902f590022\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") "
Nov 01 22:32:23.081417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.081376    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\" (UniqueName: \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\") pod \"565fc23b-8273-44b8-92c7-eb902f590022\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") "
Nov 01 22:32:23.081417 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.081396    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\") pod \"565fc23b-8273-44b8-92c7-eb902f590022\" (UID: \"565fc23b-8273-44b8-92c7-eb902f590022\") "
Nov 01 22:32:23.081904 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.081881    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca" pod "565fc23b-8273-44b8-92c7-eb902f590022" (UID: "565fc23b-8273-44b8-92c7-eb902f590022"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:23.086116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.084201    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls" pod "565fc23b-8273-44b8-92c7-eb902f590022" (UID: "565fc23b-8273-44b8-92c7-eb902f590022"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:23.086116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.084481    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5" pod "565fc23b-8273-44b8-92c7-eb902f590022" (UID: "565fc23b-8273-44b8-92c7-eb902f590022"). InnerVolumeSpecName "kube-api-access-jkbf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:23.087061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.087034    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs" pod "565fc23b-8273-44b8-92c7-eb902f590022" (UID: "565fc23b-8273-44b8-92c7-eb902f590022"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:23.130798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.130765    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:32:23.138462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.138439    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:32:23.183561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.183410    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.183561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.183446    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/565fc23b-8273-44b8-92c7-eb902f590022-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.183561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.183463    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbf5\" (UniqueName: \"kubernetes.io/projected/565fc23b-8273-44b8-92c7-eb902f590022-kube-api-access-jkbf5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.183561 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.183479    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/565fc23b-8273-44b8-92c7-eb902f590022-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.207586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.207549    5845 generic.go:356] "Generic (PLEG): container finished" podID="e07fa865-0c48-457d-8dc7-fd430c157021" containerID="0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905" exitCode=0
Nov 01 22:32:23.207759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.207599    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"
Nov 01 22:32:23.207759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.207643    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" event={"ID":"e07fa865-0c48-457d-8dc7-fd430c157021","Type":"ContainerDied","Data":"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"}
Nov 01 22:32:23.207759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.207701    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr" event={"ID":"e07fa865-0c48-457d-8dc7-fd430c157021","Type":"ContainerDied","Data":"bb33d37cf577a02b1e36af132e4e4c41d95fcf131f687e611ae310c9417f8cb0"}
Nov 01 22:32:23.207759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.207735    5845 scope.go:122] "RemoveContainer" containerID="0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"
Nov 01 22:32:23.209006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.208988    5845 generic.go:356] "Generic (PLEG): container finished" podID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" containerID="9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c" exitCode=0
Nov 01 22:32:23.209083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.209036    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" event={"ID":"330a8cb0-89af-4ce5-aa73-58e894eef2fc","Type":"ContainerDied","Data":"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"}
Nov 01 22:32:23.209083 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.209050    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"
Nov 01 22:32:23.209177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.209079    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p" event={"ID":"330a8cb0-89af-4ce5-aa73-58e894eef2fc","Type":"ContainerDied","Data":"a6be6934d6895caf718266b2ed9ca4bb4fc01970fca077eb9e64b91ad276e11a"}
Nov 01 22:32:23.211495 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.211470    5845 generic.go:356] "Generic (PLEG): container finished" podID="565fc23b-8273-44b8-92c7-eb902f590022" containerID="20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9" exitCode=0
Nov 01 22:32:23.211637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.211523    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"
Nov 01 22:32:23.211637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.211539    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" event={"ID":"565fc23b-8273-44b8-92c7-eb902f590022","Type":"ContainerDied","Data":"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"}
Nov 01 22:32:23.211637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.211575    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n" event={"ID":"565fc23b-8273-44b8-92c7-eb902f590022","Type":"ContainerDied","Data":"425c64b9cf7d76c1c1942e955c100d2f5c52c17d7e7f296c04c4ff49f9fe8283"}
Nov 01 22:32:23.218124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.218105    5845 scope.go:122] "RemoveContainer" containerID="0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"
Nov 01 22:32:23.218395 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:23.218375    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905\": container with ID starting with 0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905 not found: ID does not exist" containerID="0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"
Nov 01 22:32:23.218483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.218406    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905"} err="failed to get container status \"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905\": rpc error: code = NotFound desc = could not find container \"0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905\": container with ID starting with 0e8d872a8fed05f7f30c08a5227afa94d97503b234099c09d98ea46b8ff59905 not found: ID does not exist"
Nov 01 22:32:23.218483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.218428    5845 scope.go:122] "RemoveContainer" containerID="9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"
Nov 01 22:32:23.227034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.227019    5845 scope.go:122] "RemoveContainer" containerID="9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"
Nov 01 22:32:23.227327 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:23.227284    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c\": container with ID starting with 9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c not found: ID does not exist" containerID="9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"
Nov 01 22:32:23.227379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.227327    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c"} err="failed to get container status \"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c\": rpc error: code = NotFound desc = could not find container \"9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c\": container with ID starting with 9205e0c691cd5fc9590dead0ec071f13d2050c74b539c4c9ca3cab4646f77e3c not found: ID does not exist"
Nov 01 22:32:23.227379 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.227342    5845 scope.go:122] "RemoveContainer" containerID="20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"
Nov 01 22:32:23.236745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.236725    5845 scope.go:122] "RemoveContainer" containerID="20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"
Nov 01 22:32:23.238663 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:23.238242    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9\": container with ID starting with 20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9 not found: ID does not exist" containerID="20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"
Nov 01 22:32:23.238663 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.238324    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9"} err="failed to get container status \"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9\": rpc error: code = NotFound desc = could not find container \"20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9\": container with ID starting with 20d4d3bec8675a7218a08e553737a79c7bd543543588bae2e81f3e624d64bcd9 not found: ID does not exist"
Nov 01 22:32:23.240928 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.240909    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 22:32:23.243837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.243816    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0def36-77f9bf8467-jrl8n"]
Nov 01 22:32:23.284180 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284156    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\") pod \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") "
Nov 01 22:32:23.284288 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284245    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\" (UniqueName: \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\") pod \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\" (UID: \"330a8cb0-89af-4ce5-aa73-58e894eef2fc\") "
Nov 01 22:32:23.284374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284284    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\" (UniqueName: \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\") pod \"e07fa865-0c48-457d-8dc7-fd430c157021\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") "
Nov 01 22:32:23.284440 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284386    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\") pod \"e07fa865-0c48-457d-8dc7-fd430c157021\" (UID: \"e07fa865-0c48-457d-8dc7-fd430c157021\") "
Nov 01 22:32:23.284745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284714    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume" pod "330a8cb0-89af-4ce5-aa73-58e894eef2fc" (UID: "330a8cb0-89af-4ce5-aa73-58e894eef2fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:23.284879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.284774    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume" pod "e07fa865-0c48-457d-8dc7-fd430c157021" (UID: "e07fa865-0c48-457d-8dc7-fd430c157021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:23.285269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.285247    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07fa865-0c48-457d-8dc7-fd430c157021-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.285366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.285275    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330a8cb0-89af-4ce5-aa73-58e894eef2fc-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.286583 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.286562    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx" pod "330a8cb0-89af-4ce5-aa73-58e894eef2fc" (UID: "330a8cb0-89af-4ce5-aa73-58e894eef2fc"). InnerVolumeSpecName "kube-api-access-mdqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:23.286683 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.286620    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn" pod "e07fa865-0c48-457d-8dc7-fd430c157021" (UID: "e07fa865-0c48-457d-8dc7-fd430c157021"). InnerVolumeSpecName "kube-api-access-xjtdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:23.385989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.385954    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqmx\" (UniqueName: \"kubernetes.io/projected/330a8cb0-89af-4ce5-aa73-58e894eef2fc-kube-api-access-mdqmx\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.385989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.385986    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtdn\" (UniqueName: \"kubernetes.io/projected/e07fa865-0c48-457d-8dc7-fd430c157021-kube-api-access-xjtdn\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:23.533150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.533117    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:32:23.537476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.537454    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-56452a-649448cb74-vk8fr"]
Nov 01 22:32:23.549215 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.549183    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:32:23.558578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:23.558553    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-56452a-9f4f946f9-6vz7p"]
Nov 01 22:32:24.860819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:24.860788    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" path="/var/lib/kubelet/pods/330a8cb0-89af-4ce5-aa73-58e894eef2fc/volumes"
Nov 01 22:32:24.861316 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:24.861279    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565fc23b-8273-44b8-92c7-eb902f590022" path="/var/lib/kubelet/pods/565fc23b-8273-44b8-92c7-eb902f590022/volumes"
Nov 01 22:32:24.862189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:24.862172    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07fa865-0c48-457d-8dc7-fd430c157021" path="/var/lib/kubelet/pods/e07fa865-0c48-457d-8dc7-fd430c157021/volumes"
Nov 01 22:32:29.668974 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:29.668933    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 22:32:29.669665 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:29.669330    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy" containerID="cri-o://da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95" gracePeriod=30
Nov 01 22:32:34.823959 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:34.823917    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:32:34.935123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:34.935098    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 22:32:35.001056 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001025    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001067    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001093    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001126    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001166    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001221    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001261 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001244    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001285    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001367    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\") pod \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\" (UID: \"e71d4526-5b90-4e47-8778-b7ce2ebaace7\") "
Nov 01 22:32:35.001637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001619    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:35.001798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001639    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:35.001798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001755    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:35.001911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001824    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.001911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001833    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:35.001911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001845    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.002072 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.001952    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:35.003648 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.003612    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:32:35.003766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.003730    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:35.003812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.003794    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "kube-api-access-wvcq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:35.003882 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.003868    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy" pod "e71d4526-5b90-4e47-8778-b7ce2ebaace7" (UID: "e71d4526-5b90-4e47-8778-b7ce2ebaace7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:32:35.103137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103070    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103136    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103162    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103176    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103189    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcq7\" (UniqueName: \"kubernetes.io/projected/e71d4526-5b90-4e47-8778-b7ce2ebaace7-kube-api-access-wvcq7\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103203    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71d4526-5b90-4e47-8778-b7ce2ebaace7-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.103455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.103233    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71d4526-5b90-4e47-8778-b7ce2ebaace7-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:35.276166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.276081    5845 generic.go:356] "Generic (PLEG): container finished" podID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerID="da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95" exitCode=0
Nov 01 22:32:35.276166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.276126    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"
Nov 01 22:32:35.276390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.276172    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" event={"ID":"e71d4526-5b90-4e47-8778-b7ce2ebaace7","Type":"ContainerDied","Data":"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"}
Nov 01 22:32:35.276390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.276228    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4" event={"ID":"e71d4526-5b90-4e47-8778-b7ce2ebaace7","Type":"ContainerDied","Data":"7d2e354e5fce24423a674c0afad5e897fa8d84ff1b16bc8a9be6b8b53fb823bf"}
Nov 01 22:32:35.276390 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.276258    5845 scope.go:122] "RemoveContainer" containerID="da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"
Nov 01 22:32:35.286388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.286370    5845 scope.go:122] "RemoveContainer" containerID="da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"
Nov 01 22:32:35.286660 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:35.286638    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95\": container with ID starting with da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95 not found: ID does not exist" containerID="da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"
Nov 01 22:32:35.286746 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.286666    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95"} err="failed to get container status \"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95\": rpc error: code = NotFound desc = could not find container \"da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95\": container with ID starting with da5a573232a190f8cb32d91676966cc0ec7c06a948f4ef66260006125e093a95 not found: ID does not exist"
Nov 01 22:32:35.300187 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.300159    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 22:32:35.315236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:35.315202    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ff3e70-openshift-default-68794df669-hdcb4"]
Nov 01 22:32:36.859447 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:36.859416    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" path="/var/lib/kubelet/pods/e71d4526-5b90-4e47-8778-b7ce2ebaace7/volumes"
Nov 01 22:32:58.746668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:58.746628    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 22:32:58.747405 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:58.746879    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api" containerID="cri-o://d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e" gracePeriod=30
Nov 01 22:32:59.001024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.000966    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 22:32:59.133633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.133520    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\" (UniqueName: \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\") pod \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") "
Nov 01 22:32:59.133633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.133594    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\") pod \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") "
Nov 01 22:32:59.133633 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.133626    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\") pod \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") "
Nov 01 22:32:59.135366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.133734    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\") pod \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\" (UID: \"c5a8cdcb-ab23-41ec-a53f-c518a55319f5\") "
Nov 01 22:32:59.135366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.134924    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca" pod "c5a8cdcb-ab23-41ec-a53f-c518a55319f5" (UID: "c5a8cdcb-ab23-41ec-a53f-c518a55319f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:32:59.138757 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.138732    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8" pod "c5a8cdcb-ab23-41ec-a53f-c518a55319f5" (UID: "c5a8cdcb-ab23-41ec-a53f-c518a55319f5"). InnerVolumeSpecName "kube-api-access-d9pr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:59.138893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.138770    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs" pod "c5a8cdcb-ab23-41ec-a53f-c518a55319f5" (UID: "c5a8cdcb-ab23-41ec-a53f-c518a55319f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:59.138893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.138802    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls" pod "c5a8cdcb-ab23-41ec-a53f-c518a55319f5" (UID: "c5a8cdcb-ab23-41ec-a53f-c518a55319f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:59.234734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.234705    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:59.234734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.234732    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:59.234734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.234742    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:59.234940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.234752    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pr8\" (UniqueName: \"kubernetes.io/projected/c5a8cdcb-ab23-41ec-a53f-c518a55319f5-kube-api-access-d9pr8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:59.396627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.396546    5845 generic.go:356] "Generic (PLEG): container finished" podID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerID="d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e" exitCode=0
Nov 01 22:32:59.396627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.396594    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"
Nov 01 22:32:59.396627 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.396619    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" event={"ID":"c5a8cdcb-ab23-41ec-a53f-c518a55319f5","Type":"ContainerDied","Data":"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"}
Nov 01 22:32:59.396894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.396663    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt" event={"ID":"c5a8cdcb-ab23-41ec-a53f-c518a55319f5","Type":"ContainerDied","Data":"ea3995e6bee4b300662c8fa0d5970ef831b272248fca5490227eec651ed6b57b"}
Nov 01 22:32:59.396894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.396683    5845 scope.go:122] "RemoveContainer" containerID="d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"
Nov 01 22:32:59.412024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.412006    5845 scope.go:122] "RemoveContainer" containerID="d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"
Nov 01 22:32:59.412358 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:32:59.412327    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e\": container with ID starting with d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e not found: ID does not exist" containerID="d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"
Nov 01 22:32:59.412436 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.412367    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e"} err="failed to get container status \"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e\": rpc error: code = NotFound desc = could not find container \"d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e\": container with ID starting with d1ea063a3d79b565364ae29557da8e28bf032aa39c08da3746f49e78799c9d6e not found: ID does not exist"
Nov 01 22:32:59.419250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.419227    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 22:32:59.432321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:32:59.429281    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1f3a6b-58b57f6c8-nltdt"]
Nov 01 22:33:00.859847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:00.859812    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" path="/var/lib/kubelet/pods/c5a8cdcb-ab23-41ec-a53f-c518a55319f5/volumes"
Nov 01 22:33:04.817246 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:33:04.817206    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:33:34.818473 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:33:34.818433    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:33:46.772512 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772478    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"]
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772897    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772909    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772925    5845 cpu_manager.go:414] "containerMap: removing container" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772929    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772939    5845 cpu_manager.go:414] "containerMap: removing container" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772944    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772950    5845 cpu_manager.go:414] "containerMap: removing container" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772955    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772967    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772975    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772981    5845 cpu_manager.go:414] "containerMap: removing container" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772986    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772994    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.772998    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773005    5845 cpu_manager.go:414] "containerMap: removing container" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy"
Nov 01 22:33:46.773003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773009    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773016    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e07fa865-0c48-457d-8dc7-fd430c157021" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773020    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07fa865-0c48-457d-8dc7-fd430c157021" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773027    5845 cpu_manager.go:414] "containerMap: removing container" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773033    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773040    5845 cpu_manager.go:414] "containerMap: removing container" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773047    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773061    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773065    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773073    5845 cpu_manager.go:414] "containerMap: removing container" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773078    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773085    5845 cpu_manager.go:414] "containerMap: removing container" podUID="1dfdb033-56bf-41e6-93df-e8a66e3de202" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773090    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfdb033-56bf-41e6-93df-e8a66e3de202" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773098    5845 cpu_manager.go:414] "containerMap: removing container" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773102    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773109    5845 cpu_manager.go:414] "containerMap: removing container" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773113    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773123    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773131    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773141    5845 cpu_manager.go:414] "containerMap: removing container" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773146    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773152    5845 cpu_manager.go:414] "containerMap: removing container" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773156    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773162    5845 cpu_manager.go:414] "containerMap: removing container" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773167    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773174    5845 cpu_manager.go:414] "containerMap: removing container" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773178    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773191    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773197    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773215    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773223    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773232    5845 cpu_manager.go:414] "containerMap: removing container" podUID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773237    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773246    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773252    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773263    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773270    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773277    5845 cpu_manager.go:414] "containerMap: removing container" podUID="18231c84-9262-4ab2-949e-968665edcbf1" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773284    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18231c84-9262-4ab2-949e-968665edcbf1" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773313    5845 cpu_manager.go:414] "containerMap: removing container" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773321    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773334    5845 cpu_manager.go:414] "containerMap: removing container" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773342    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773349    5845 cpu_manager.go:414] "containerMap: removing container" podUID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773353    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773362    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773369    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773381    5845 cpu_manager.go:414] "containerMap: removing container" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773387    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773400    5845 cpu_manager.go:414] "containerMap: removing container" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773407    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773421    5845 cpu_manager.go:414] "containerMap: removing container" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773427    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773442    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773449    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773459    5845 cpu_manager.go:414] "containerMap: removing container" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773465    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773475    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773483    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773494    5845 cpu_manager.go:414] "containerMap: removing container" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773501    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773510    5845 cpu_manager.go:414] "containerMap: removing container" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773517    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773528    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773534    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773546    5845 cpu_manager.go:414] "containerMap: removing container" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773553    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773568    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773575    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773587    5845 cpu_manager.go:414] "containerMap: removing container" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773594    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773605    5845 cpu_manager.go:414] "containerMap: removing container" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773612    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773624    5845 cpu_manager.go:414] "containerMap: removing container" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773631    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773639    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773646    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773657    5845 cpu_manager.go:414] "containerMap: removing container" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773665    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773775    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b65487-1cd9-4053-bb63-f50c185beb79" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773786    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773797    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="565fc23b-8273-44b8-92c7-eb902f590022" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773807    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c092b7ee-3175-4c71-8df8-d69b6a6f25d3" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773814    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71d4526-5b90-4e47-8778-b7ce2ebaace7" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773824    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b6f384-447a-44d8-b2a0-dc547696f39b" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773833    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d145ee6-4342-41e4-a95b-9365770eecf7" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773841    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="039aafea-77e9-4db4-9e2c-faf5abf0779f" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773850    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773859    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="d832d3b8-c3e0-4d1e-a44c-e80aca53a364" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773868    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f123a7c8-5b58-4630-8eb1-4984e30e9cfb" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773877    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db1449e-00fb-409c-8a1f-2a06ad3f94d5" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773886    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773895    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af9a875-0b45-4e56-bb74-9ea2ff7f53a8" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773902    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07fa865-0c48-457d-8dc7-fd430c157021" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773911    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7bbaad-d051-4782-a918-386d7f69bd6d" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773920    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="d443e8ab-a261-40e1-9884-bbeee643fc4c" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773928    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c93af5-3488-4d83-960b-ead7d2f593e4" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773938    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773946    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfdb033-56bf-41e6-93df-e8a66e3de202" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773956    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773964    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773971    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="6625bced-23e1-4099-9b61-625eb9f9e05f" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773979    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773989    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="614512c6-d381-48ee-845c-fc429ba794b9" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.773999    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="b204b3aa-d4fe-48b1-93b7-888db9d93a20" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774006    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="426bc111-0f0a-4c63-a50a-519e47476539" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774013    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774024    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a8cdcb-ab23-41ec-a53f-c518a55319f5" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774034    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3574beae-8658-4d85-9d12-255ade49f917" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774043    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="330a8cb0-89af-4ce5-aa73-58e894eef2fc" containerName="payload-pre-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774052    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3e2e23-6f3d-48cb-9e35-574acedff5e0" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774061    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774070    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="18231c84-9262-4ab2-949e-968665edcbf1" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774337    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774353    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="f934630d-e314-4d71-90f1-539f20f97241" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774359    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="62b0fab0-718a-497e-881d-70a837997119" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774366    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d36006-6cbe-4f59-aac1-5b1302e5beb2" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774376    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbe1608-dc84-42dd-83b7-54be275ae505" containerName="payload-processing"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774383    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89872b0-e9f9-4e00-9dbc-c8b2ca3fa3c6" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774389    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774397    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d4bcd7-f7ed-4f73-95a0-235581345bec" containerName="istio-proxy"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774407    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e36822e-f396-4ebe-a7b9-0b3a967446c6" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774414    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774424    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4cc922-e1aa-4edf-bc11-26809fd8cd10" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774433    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="9240b129-f08e-44df-9f42-da8c7cae75ef" containerName="maas-api"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774441    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb43ab1-4a4a-4d10-8245-7c0d275d2813" containerName="main"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774451    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:33:46.774619 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.774461    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9a1de9-e72f-488f-8c08-83a4de512f2e" containerName="istio-proxy"
Nov 01 22:33:46.780277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.777781    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.791657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.791624    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"]
Nov 01 22:33:46.903310 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903263    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903417    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903522 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903495    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903561    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903672 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903649    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn42l\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-kube-api-access-fn42l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903690    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903731    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903778 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903755    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:46.903918 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:46.903832    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004435 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004397    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004444    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004477    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004514    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004564    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004618 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004592    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004645    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004686    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn42l\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-kube-api-access-fn42l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.004960 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.004716    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.005174 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.005148    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.005240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.005169    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.005240 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.005227    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.005411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.005394    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.005563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.005543    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.007442 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.007421    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.007552 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.007535    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.016774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.016746    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.017098 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.017075    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn42l\" (UniqueName: \"kubernetes.io/projected/b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec-kube-api-access-fn42l\") pod \"maas-default-gateway-openshift-default-5bc884dd87-prd6f\" (UID: \"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.098022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.097920    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:47.382994 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.382963    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"]
Nov 01 22:33:47.386060 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:33:47.386033    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81dfdb2_ef94_4de7_9b0a_b8de7d5a09ec.slice/crio-89f206f4b86af1d594d99e0b6683b3dd29a4e7066853db6df74cfcb8bc4ff0d9 WatchSource:0}: Error finding container 89f206f4b86af1d594d99e0b6683b3dd29a4e7066853db6df74cfcb8bc4ff0d9: Status 404 returned error can't find the container with id 89f206f4b86af1d594d99e0b6683b3dd29a4e7066853db6df74cfcb8bc4ff0d9
Nov 01 22:33:47.388326 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.388263    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:47.388446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.388375    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:47.388446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.388422    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:47.632914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.632792    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" event={"ID":"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec","Type":"ContainerStarted","Data":"e88bf57edb9643a7e5ec81ce56cb59d47fb9f2f9943bfda4330dd0780796396b"}
Nov 01 22:33:47.632914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.632876    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" event={"ID":"b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec","Type":"ContainerStarted","Data":"89f206f4b86af1d594d99e0b6683b3dd29a4e7066853db6df74cfcb8bc4ff0d9"}
Nov 01 22:33:47.657466 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:47.657400    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podStartSLOduration=1.657381305 podStartE2EDuration="1.657381305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:47.656070693 +0000 UTC m=+2930.912250454" watchObservedRunningTime="2026-11-01 22:33:47.657381305 +0000 UTC m=+2930.913561069"
Nov 01 22:33:48.098989 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:48.098958    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:48.100514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:48.100488    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:48.100621 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:48.100533    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:49.099073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:49.099033    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:49.099505 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:49.099102    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:50.098593 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:50.098555    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:50.098808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:50.098613    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:51.098737 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:51.098694    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:51.099168 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:51.098754    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:52.099098 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:52.099060    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:52.099569 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:52.099117    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:53.098676 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:53.098638    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:53.098840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:53.098696    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:54.098280 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:54.098242    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-prd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 01 22:33:54.098707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:54.098308    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f" podUID="b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 01 22:33:55.592919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:55.592881    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:55.593420 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:55.593232    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:33:55.594196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:33:55.594171    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-prd6f"
Nov 01 22:34:04.815759 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:34:04.815722    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:34:34.828931 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:34:34.828883    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:34:46.831756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.831720    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"]
Nov 01 22:34:46.832422 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832401    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:34:46.832518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832423    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366d1dd-7a69-47c6-983f-fb72ddccce54" containerName="istio-proxy"
Nov 01 22:34:46.832518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832456    5845 cpu_manager.go:414] "containerMap: removing container" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:34:46.832518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832464    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b88531-bad5-465a-8275-01770308a874" containerName="cleanup"
Nov 01 22:34:46.832518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832508    5845 cpu_manager.go:414] "containerMap: removing container" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:34:46.832518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832516    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a05e232c-d130-4b79-a95a-009bc1492770" containerName="cleanup"
Nov 01 22:34:46.832786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832621    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="36470550-abf4-47f8-8c5e-583fc7e875d5" containerName="istio-proxy"
Nov 01 22:34:46.832786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.832641    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="4042e0fa-32fa-46cc-8d90-7b13742efcf5" containerName="istio-proxy"
Nov 01 22:34:46.835858 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.835828    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.850642 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.850612    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"]
Nov 01 22:34:46.918392 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918363    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.918555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918415    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.918555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918474    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.918555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918499    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fb2f07-f693-45bc-90e5-f622fdef72e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.918687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918580    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.918731 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918702    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xvw\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-kube-api-access-p4xvw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.919059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918869    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.919059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918924    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:46.919059 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:46.918962    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.019752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019715    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xvw\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-kube-api-access-p4xvw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.019896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019780    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.019896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019809    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.019896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019838    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.019896 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019864    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.019909    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020040    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020093    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fb2f07-f693-45bc-90e5-f622fdef72e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020152    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020251    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020490 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020425    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020515    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020565    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.020811 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.020790    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72fb2f07-f693-45bc-90e5-f622fdef72e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.022182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.022158    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.022437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.022419    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.031606 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.031586    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.031739 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.031716    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xvw\" (UniqueName: \"kubernetes.io/projected/72fb2f07-f693-45bc-90e5-f622fdef72e5-kube-api-access-p4xvw\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fp2b7\" (UID: \"72fb2f07-f693-45bc-90e5-f622fdef72e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.151137 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.151053    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:47.433860 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.433832    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"]
Nov 01 22:34:47.437196 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:34:47.437146    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fb2f07_f693_45bc_90e5_f622fdef72e5.slice/crio-cfbed29ce4ec072a4fbd1c10ed96873a5d5db2d59c57c5e588d90919992c3542 WatchSource:0}: Error finding container cfbed29ce4ec072a4fbd1c10ed96873a5d5db2d59c57c5e588d90919992c3542: Status 404 returned error can't find the container with id cfbed29ce4ec072a4fbd1c10ed96873a5d5db2d59c57c5e588d90919992c3542
Nov 01 22:34:47.439652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.439605    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:47.439764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.439695    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:47.439764 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.439737    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:47.977427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.977383    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" event={"ID":"72fb2f07-f693-45bc-90e5-f622fdef72e5","Type":"ContainerStarted","Data":"280559c6bea8378f268ed4f2973a95bd142b1d4685598066e3b697e3873264c3"}
Nov 01 22:34:47.977866 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:47.977448    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" event={"ID":"72fb2f07-f693-45bc-90e5-f622fdef72e5","Type":"ContainerStarted","Data":"cfbed29ce4ec072a4fbd1c10ed96873a5d5db2d59c57c5e588d90919992c3542"}
Nov 01 22:34:48.002455 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:48.002405    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podStartSLOduration=2.002392402 podStartE2EDuration="2.002392402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:48.001499634 +0000 UTC m=+2991.257679393" watchObservedRunningTime="2026-11-01 22:34:48.002392402 +0000 UTC m=+2991.258572165"
Nov 01 22:34:48.164332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:48.160666    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:34:48.168320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:48.164577    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:48.168320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:48.164638    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:49.160579 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:49.160543    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:49.161177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:49.161134    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:50.156322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:50.152650    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:50.156322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:50.152714    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:51.164322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:51.162781    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:51.164322 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:51.162877    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:52.163321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:52.163049    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:52.163321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:52.163125    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:53.078518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.078479    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 22:34:53.079024 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.078664    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery" containerID="cri-o://14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040" gracePeriod=30
Nov 01 22:34:53.155651 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.155606    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:53.155933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.155905    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:53.352981 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.352957    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 22:34:53.414631 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414589    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.414813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414678    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.414813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414739    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.414813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414796    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.415003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414855    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.415003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414911    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.415003 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.414953    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\") pod \"31852f01-d286-4d4d-b3cd-1819d347bf60\" (UID: \"31852f01-d286-4d4d-b3cd-1819d347bf60\") "
Nov 01 22:34:53.415186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.415157    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:34:53.415348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.415325    5845 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-ca-configmap\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.417407 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417379    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "kube-api-access-cjn7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:34:53.417514 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417456    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:34:53.417712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417686    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:34:53.417808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417711    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:34:53.417808 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417769    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:34:53.417972 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.417947    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts" pod "31852f01-d286-4d4d-b3cd-1819d347bf60" (UID: "31852f01-d286-4d4d-b3cd-1819d347bf60"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:34:53.516425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516398    5845 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-csr-dns-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.516425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516424    5845 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-cacerts\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.516425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516435    5845 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-kubeconfig\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.516643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516444    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjn7n\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-kube-api-access-cjn7n\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.516643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516453    5845 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/31852f01-d286-4d4d-b3cd-1819d347bf60-local-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:53.516643 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:53.516462    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31852f01-d286-4d4d-b3cd-1819d347bf60-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:34:54.010282 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.010240    5845 generic.go:356] "Generic (PLEG): container finished" podID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerID="14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040" exitCode=0
Nov 01 22:34:54.010509 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.010360    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"
Nov 01 22:34:54.010625 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.010265    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" event={"ID":"31852f01-d286-4d4d-b3cd-1819d347bf60","Type":"ContainerDied","Data":"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"}
Nov 01 22:34:54.010752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.010667    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp" event={"ID":"31852f01-d286-4d4d-b3cd-1819d347bf60","Type":"ContainerDied","Data":"c13405c57dc5b28c970a470dcb17706acf3850cf9e79cd29a0c70ad203aa627c"}
Nov 01 22:34:54.010752 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.010700    5845 scope.go:122] "RemoveContainer" containerID="14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"
Nov 01 22:34:54.024842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.024817    5845 scope.go:122] "RemoveContainer" containerID="14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"
Nov 01 22:34:54.025187 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:34:54.025154    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040\": container with ID starting with 14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040 not found: ID does not exist" containerID="14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"
Nov 01 22:34:54.025271 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.025203    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040"} err="failed to get container status \"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040\": rpc error: code = NotFound desc = could not find container \"14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040\": container with ID starting with 14850be173790a4c49ef00e43ba54298d833e9ebe7e964da83f437767c0e7040 not found: ID does not exist"
Nov 01 22:34:54.036788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.036760    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 22:34:54.040582 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.040561    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dsdmp"]
Nov 01 22:34:54.151519 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.151484    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 01 22:34:54.151933 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.151538    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 01 22:34:54.860113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:54.860083    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" path="/var/lib/kubelet/pods/31852f01-d286-4d4d-b3cd-1819d347bf60/volumes"
Nov 01 22:34:56.152784 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:56.152735    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:34:56.153206 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:56.152821    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:34:57.211483 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:57.211427    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:34:57.212011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:57.211525    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:34:58.213654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:58.213607    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:34:58.214188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:58.213671    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:34:59.254484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.254438    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:34:59.254971 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.254521    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:34:59.332730 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.332698    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:34:59.337536 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.337516    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:34:59.354446 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.354425    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:34:59.360100 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:34:59.360081    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:35:00.255437 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:35:00.255393    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:35:00.255871 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:35:00.255469    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:35:00.482293 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:35:00.482257    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:35:00.482735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:35:00.482711    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:35:00.483675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:35:00.483657    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7"
Nov 01 22:35:04.816235 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:35:04.816188    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:35:34.815141 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:35:34.815103    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:36:04.842958 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:36:04.842921    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:36:34.817229 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:36:34.817186    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:37:04.815505 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:37:04.815468    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:37:34.817161 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:37:34.817123    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:38:04.815060 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:38:04.815022    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:38:34.815108 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:38:34.815069    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:39:04.827720 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:39:04.827630    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:39:15.354362 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.354227    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:39:15.355057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.354688    5845 cpu_manager.go:414] "containerMap: removing container" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery"
Nov 01 22:39:15.355057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.354702    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery"
Nov 01 22:39:15.355057 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.354800    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="31852f01-d286-4d4d-b3cd-1819d347bf60" containerName="discovery"
Nov 01 22:39:15.358538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.358518    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.368277 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.368252    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-096081-serving-cert\""
Nov 01 22:39:15.371956 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.371927    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:39:15.421229 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.421195    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlmwj\" (UniqueName: \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.421432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.421233    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.421432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.421313    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.421432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.421366    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.522750 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.522721    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlmwj\" (UniqueName: \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.522947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.522756    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.522947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.522784    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.522947 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.522848    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.523542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.523522    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.523747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.523722    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:39:15.526013 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.525983    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.526121 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.526019    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.528307 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.528278    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.537786 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.537760    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:39:15.540279 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.540255    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlmwj\" (UniqueName: \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\") pod \"maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.553704 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.553680    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fc5cbdbf-dockercfg-pwlkj\""
Nov 01 22:39:15.558838 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.558819    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-096081\""
Nov 01 22:39:15.624576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.624481    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdlf\" (UniqueName: \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.624759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.624654    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.673856 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.673824    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:15.726009 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.725964    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdlf\" (UniqueName: \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.726193 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.726157    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.727016 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.726987    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.745253 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.745194    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdlf\" (UniqueName: \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\") pod \"payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.866874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.866842    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:15.959073 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.959044    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:39:15.964427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:15.964406    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:39:16.152115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.152069    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:39:16.395412 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.395376    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" event={"ID":"dce1c8a9-cae1-428d-b921-3eed7236337b","Type":"ContainerStarted","Data":"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"}
Nov 01 22:39:16.395897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.395440    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" event={"ID":"dce1c8a9-cae1-428d-b921-3eed7236337b","Type":"ContainerStarted","Data":"8efd6794c15ca5e5d80889b2c6884abb774fcc3edb052c8aecc8cf93536e6942"}
Nov 01 22:39:16.395897 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.395568    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:16.396747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.396691    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" event={"ID":"e26e9aee-51a6-4584-9b71-7e7671452384","Type":"ContainerStarted","Data":"0e29b8f58b765845daca6d0c992468eb9614340025f3deea6043592c8a3a5bae"}
Nov 01 22:39:16.417888 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:16.417845    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" podStartSLOduration=1.417832501 podStartE2EDuration="1.417832501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:16.415582734 +0000 UTC m=+3259.671762496" watchObservedRunningTime="2026-11-01 22:39:16.417832501 +0000 UTC m=+3259.674012290"
Nov 01 22:39:17.404220 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:17.404180    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" event={"ID":"e26e9aee-51a6-4584-9b71-7e7671452384","Type":"ContainerStarted","Data":"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"}
Nov 01 22:39:17.428233 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:17.428161    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" podStartSLOduration=1.95813437 podStartE2EDuration="2.428144245s" totalImagesPullingTime="470.009875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:17.426021289 +0000 UTC m=+3260.682201052" watchObservedRunningTime="2026-11-01 22:39:17.428144245 +0000 UTC m=+3260.684324011"
Nov 01 22:39:18.408607 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:18.408574    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:24.418713 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:24.418671    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:39:27.407128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:27.407095    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:39:34.816178 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:39:34.816140    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:39:59.365332 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:59.365290    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:39:59.373387 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:59.373363    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:39:59.393571 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:59.393550    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:39:59.398040 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:39:59.398021    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:40:04.815437 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:40:04.815393    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:40:24.357879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:24.357847    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:40:25.047553 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.047513    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:40:25.047768 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.047739    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" podUID="e26e9aee-51a6-4584-9b71-7e7671452384" containerName="maas-api" containerID="cri-o://ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc" gracePeriod=30
Nov 01 22:40:25.310660 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.310600    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:40:25.316354 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.316331    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\" (UniqueName: \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\") pod \"e26e9aee-51a6-4584-9b71-7e7671452384\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") "
Nov 01 22:40:25.316470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.316409    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\") pod \"e26e9aee-51a6-4584-9b71-7e7671452384\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") "
Nov 01 22:40:25.316470 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.316461    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\") pod \"e26e9aee-51a6-4584-9b71-7e7671452384\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") "
Nov 01 22:40:25.316609 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.316489    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\") pod \"e26e9aee-51a6-4584-9b71-7e7671452384\" (UID: \"e26e9aee-51a6-4584-9b71-7e7671452384\") "
Nov 01 22:40:25.316899 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.316862    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca" pod "e26e9aee-51a6-4584-9b71-7e7671452384" (UID: "e26e9aee-51a6-4584-9b71-7e7671452384"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:25.318410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.318381    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls" pod "e26e9aee-51a6-4584-9b71-7e7671452384" (UID: "e26e9aee-51a6-4584-9b71-7e7671452384"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:25.318510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.318438    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs" pod "e26e9aee-51a6-4584-9b71-7e7671452384" (UID: "e26e9aee-51a6-4584-9b71-7e7671452384"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:25.318510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.318460    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj" pod "e26e9aee-51a6-4584-9b71-7e7671452384" (UID: "e26e9aee-51a6-4584-9b71-7e7671452384"). InnerVolumeSpecName "kube-api-access-vlmwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:25.418026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.417990    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:25.418026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.418020    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26e9aee-51a6-4584-9b71-7e7671452384-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:25.418026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.418031    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26e9aee-51a6-4584-9b71-7e7671452384-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:25.418500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.418040    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlmwj\" (UniqueName: \"kubernetes.io/projected/e26e9aee-51a6-4584-9b71-7e7671452384-kube-api-access-vlmwj\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:25.731699 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.731649    5845 generic.go:356] "Generic (PLEG): container finished" podID="e26e9aee-51a6-4584-9b71-7e7671452384" containerID="ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc" exitCode=0
Nov 01 22:40:25.731878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.731708    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"
Nov 01 22:40:25.731878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.731735    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" event={"ID":"e26e9aee-51a6-4584-9b71-7e7671452384","Type":"ContainerDied","Data":"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"}
Nov 01 22:40:25.731878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.731810    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55" event={"ID":"e26e9aee-51a6-4584-9b71-7e7671452384","Type":"ContainerDied","Data":"0e29b8f58b765845daca6d0c992468eb9614340025f3deea6043592c8a3a5bae"}
Nov 01 22:40:25.731878 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.731832    5845 scope.go:122] "RemoveContainer" containerID="ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"
Nov 01 22:40:25.741644 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.741627    5845 scope.go:122] "RemoveContainer" containerID="ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"
Nov 01 22:40:25.741887 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:40:25.741867    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc\": container with ID starting with ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc not found: ID does not exist" containerID="ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"
Nov 01 22:40:25.741940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.741895    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc"} err="failed to get container status \"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc\": rpc error: code = NotFound desc = could not find container \"ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc\": container with ID starting with ccf53284a95298da96375032280db689f8120094603fc9a6cc1084a3072e2ffc not found: ID does not exist"
Nov 01 22:40:25.753475 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.753454    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:40:25.758010 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:25.757989    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-096081-679f966fbd-hpb55"]
Nov 01 22:40:26.860147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:26.860116    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26e9aee-51a6-4584-9b71-7e7671452384" path="/var/lib/kubelet/pods/e26e9aee-51a6-4584-9b71-7e7671452384/volumes"
Nov 01 22:40:29.358411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:29.358372    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing" containerID="cri-o://581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe" gracePeriod=25
Nov 01 22:40:34.814698 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:40:34.814658    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:40:37.405879 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:37.405841    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:40:46.542473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:46.542424    5845 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fp2b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:40:46.543091 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:46.542481    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fp2b7" podUID="72fb2f07-f693-45bc-90e5-f622fdef72e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:40:47.406812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:47.406775    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:40:49.613835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.613807    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:40:49.764563 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.764533    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\") pod \"dce1c8a9-cae1-428d-b921-3eed7236337b\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") "
Nov 01 22:40:49.764747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.764593    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\" (UniqueName: \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\") pod \"dce1c8a9-cae1-428d-b921-3eed7236337b\" (UID: \"dce1c8a9-cae1-428d-b921-3eed7236337b\") "
Nov 01 22:40:49.764917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.764895    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume" pod "dce1c8a9-cae1-428d-b921-3eed7236337b" (UID: "dce1c8a9-cae1-428d-b921-3eed7236337b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:49.766721 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.766700    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf" pod "dce1c8a9-cae1-428d-b921-3eed7236337b" (UID: "dce1c8a9-cae1-428d-b921-3eed7236337b"). InnerVolumeSpecName "kube-api-access-zrdlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:49.859377 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.859346    5845 generic.go:356] "Generic (PLEG): container finished" podID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerID="581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe" exitCode=0
Nov 01 22:40:49.859565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.859392    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"
Nov 01 22:40:49.859565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.859436    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" event={"ID":"dce1c8a9-cae1-428d-b921-3eed7236337b","Type":"ContainerDied","Data":"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"}
Nov 01 22:40:49.859565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.859491    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7" event={"ID":"dce1c8a9-cae1-428d-b921-3eed7236337b","Type":"ContainerDied","Data":"8efd6794c15ca5e5d80889b2c6884abb774fcc3edb052c8aecc8cf93536e6942"}
Nov 01 22:40:49.859565 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.859519    5845 scope.go:122] "RemoveContainer" containerID="581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"
Nov 01 22:40:49.865400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.865280    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce1c8a9-cae1-428d-b921-3eed7236337b-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:49.865400 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.865333    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdlf\" (UniqueName: \"kubernetes.io/projected/dce1c8a9-cae1-428d-b921-3eed7236337b-kube-api-access-zrdlf\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:49.869120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.869102    5845 scope.go:122] "RemoveContainer" containerID="581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"
Nov 01 22:40:49.869388 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:40:49.869370    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe\": container with ID starting with 581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe not found: ID does not exist" containerID="581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"
Nov 01 22:40:49.869448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.869397    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe"} err="failed to get container status \"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe\": rpc error: code = NotFound desc = could not find container \"581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe\": container with ID starting with 581abfd5b6e4071fe1bae5c786712c397c3a9a4b6f44c2b337fa60db07360bbe not found: ID does not exist"
Nov 01 22:40:49.882598 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.882578    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:40:49.888117 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:49.888093    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-096081-74b89c648hcqp7"]
Nov 01 22:40:50.860639 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:40:50.860605    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" path="/var/lib/kubelet/pods/dce1c8a9-cae1-428d-b921-3eed7236337b/volumes"
Nov 01 22:41:04.816261 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:41:04.816222    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:41:34.814632 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:41:34.814594    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:42:04.816337 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:42:04.816275    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:42:34.815160 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:42:34.815125    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:42:52.673188 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673154    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:42:52.673832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673799    5845 cpu_manager.go:414] "containerMap: removing container" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing"
Nov 01 22:42:52.673832 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673820    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing"
Nov 01 22:42:52.673961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673866    5845 cpu_manager.go:414] "containerMap: removing container" podUID="e26e9aee-51a6-4584-9b71-7e7671452384" containerName="maas-api"
Nov 01 22:42:52.673961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673875    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26e9aee-51a6-4584-9b71-7e7671452384" containerName="maas-api"
Nov 01 22:42:52.674064 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.673977    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce1c8a9-cae1-428d-b921-3eed7236337b" containerName="payload-processing"
Nov 01 22:42:52.674064 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.674007    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26e9aee-51a6-4584-9b71-7e7671452384" containerName="maas-api"
Nov 01 22:42:52.677736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.677715    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.688366 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.688336    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3813a1-serving-cert\""
Nov 01 22:42:52.692686 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.691404    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:42:52.824338 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.824309    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:42:52.826735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.826707    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.826876 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.826758    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.826925 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.826884    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tzk\" (UniqueName: \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.826967 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.826953    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.828082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.828068    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:52.837915 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.837887    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:42:52.838026 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.837944    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5ea6d650-dockercfg-ddkj2\""
Nov 01 22:42:52.859914 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.859891    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3813a1\""
Nov 01 22:42:52.927745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927645    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.927745 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927717    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:52.927952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927757    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7d2\" (UniqueName: \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:52.927952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927859    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.927952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927918    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.928081 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.927977    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tzk\" (UniqueName: \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.928539 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.928516    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.930199 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.930177    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.930359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.930338    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.940214 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.940185    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tzk\" (UniqueName: \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\") pod \"maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:52.993806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:52.993783    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:53.028936 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.028907    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7d2\" (UniqueName: \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.029066 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.029054    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.029717 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.029698    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.043450 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.043377    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7d2\" (UniqueName: \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\") pod \"payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.145109 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.145084    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.275163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.275126    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:42:53.279749 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:42:53.279711    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0026ec_6f2b_4b00_b0d8_4fe060780adf.slice/crio-6f9e0f9d5e99411a8d8e5e1385c4ac66c5890b6edc4691f896a52274bc47f7b4 WatchSource:0}: Error finding container 6f9e0f9d5e99411a8d8e5e1385c4ac66c5890b6edc4691f896a52274bc47f7b4: Status 404 returned error can't find the container with id 6f9e0f9d5e99411a8d8e5e1385c4ac66c5890b6edc4691f896a52274bc47f7b4
Nov 01 22:42:53.421998 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.421908    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:42:53.424727 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:42:53.424701    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedec63e_7e14_4617_b95a_9e09489f982b.slice/crio-498105331c8561015dcdd241571a01b2a013943ce8bea32a7db954a61f1872d6 WatchSource:0}: Error finding container 498105331c8561015dcdd241571a01b2a013943ce8bea32a7db954a61f1872d6: Status 404 returned error can't find the container with id 498105331c8561015dcdd241571a01b2a013943ce8bea32a7db954a61f1872d6
Nov 01 22:42:53.484599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.484570    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" event={"ID":"0a0026ec-6f2b-4b00-b0d8-4fe060780adf","Type":"ContainerStarted","Data":"6f9e0f9d5e99411a8d8e5e1385c4ac66c5890b6edc4691f896a52274bc47f7b4"}
Nov 01 22:42:53.486065 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.486037    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" event={"ID":"fedec63e-7e14-4617-b95a-9e09489f982b","Type":"ContainerStarted","Data":"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"}
Nov 01 22:42:53.486189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.486091    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" event={"ID":"fedec63e-7e14-4617-b95a-9e09489f982b","Type":"ContainerStarted","Data":"498105331c8561015dcdd241571a01b2a013943ce8bea32a7db954a61f1872d6"}
Nov 01 22:42:53.486189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.486181    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:42:53.505946 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:53.505891    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" podStartSLOduration=1.505874123 podStartE2EDuration="1.505874123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:53.504778852 +0000 UTC m=+3476.760958617" watchObservedRunningTime="2026-11-01 22:42:53.505874123 +0000 UTC m=+3476.762053886"
Nov 01 22:42:54.493092 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:54.493054    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" event={"ID":"0a0026ec-6f2b-4b00-b0d8-4fe060780adf","Type":"ContainerStarted","Data":"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"}
Nov 01 22:42:54.493617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:54.493312    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:42:54.514241 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:42:54.514195    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" podStartSLOduration=2.103977487 podStartE2EDuration="2.514183501s" totalImagesPullingTime="410.206014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:54.513812686 +0000 UTC m=+3477.769992451" watchObservedRunningTime="2026-11-01 22:42:54.514183501 +0000 UTC m=+3477.770363265"
Nov 01 22:43:00.504893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:43:00.504863    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:43:04.496439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:43:04.496406    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:43:04.815274 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:43:04.815196    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:43:34.824198 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:43:34.824155    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:43:59.474943 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:43:59.474907    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:43:59.639256 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:43:59.639216    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:43:59.639542 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:43:59.639507    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" podUID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" containerName="maas-api" containerID="cri-o://df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988" gracePeriod=30
Nov 01 22:44:00.101843 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.101817    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:44:00.200211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200130    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\") pod \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") "
Nov 01 22:44:00.200211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200168    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\") pod \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") "
Nov 01 22:44:00.200211 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200197    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\" (UniqueName: \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\") pod \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") "
Nov 01 22:44:00.200500 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200410    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\") pod \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\" (UID: \"0a0026ec-6f2b-4b00-b0d8-4fe060780adf\") "
Nov 01 22:44:00.200575 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200517    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca" pod "0a0026ec-6f2b-4b00-b0d8-4fe060780adf" (UID: "0a0026ec-6f2b-4b00-b0d8-4fe060780adf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:00.200779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.200759    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:00.202484 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.202457    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs" pod "0a0026ec-6f2b-4b00-b0d8-4fe060780adf" (UID: "0a0026ec-6f2b-4b00-b0d8-4fe060780adf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:00.202593 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.202501    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls" pod "0a0026ec-6f2b-4b00-b0d8-4fe060780adf" (UID: "0a0026ec-6f2b-4b00-b0d8-4fe060780adf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:00.202593 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.202512    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk" pod "0a0026ec-6f2b-4b00-b0d8-4fe060780adf" (UID: "0a0026ec-6f2b-4b00-b0d8-4fe060780adf"). InnerVolumeSpecName "kube-api-access-f7tzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:00.301258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.301229    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:00.301258 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.301255    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tzk\" (UniqueName: \"kubernetes.io/projected/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-kube-api-access-f7tzk\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:00.301438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.301267    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0026ec-6f2b-4b00-b0d8-4fe060780adf-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:00.834836 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.834800    5845 generic.go:356] "Generic (PLEG): container finished" podID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" containerID="df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988" exitCode=0
Nov 01 22:44:00.835250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.834842    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"
Nov 01 22:44:00.835250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.834858    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" event={"ID":"0a0026ec-6f2b-4b00-b0d8-4fe060780adf","Type":"ContainerDied","Data":"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"}
Nov 01 22:44:00.835250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.834921    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2" event={"ID":"0a0026ec-6f2b-4b00-b0d8-4fe060780adf","Type":"ContainerDied","Data":"6f9e0f9d5e99411a8d8e5e1385c4ac66c5890b6edc4691f896a52274bc47f7b4"}
Nov 01 22:44:00.835250 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.834946    5845 scope.go:122] "RemoveContainer" containerID="df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"
Nov 01 22:44:00.845077 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.845059    5845 scope.go:122] "RemoveContainer" containerID="df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"
Nov 01 22:44:00.845378 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:44:00.845352    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988\": container with ID starting with df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988 not found: ID does not exist" containerID="df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"
Nov 01 22:44:00.845443 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.845389    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988"} err="failed to get container status \"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988\": rpc error: code = NotFound desc = could not find container \"df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988\": container with ID starting with df09787b12bd67b1eac17dfdddc5209674fd61525c14f07d8affc5e0b13d4988 not found: ID does not exist"
Nov 01 22:44:00.860176 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.860146    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:44:00.865880 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:00.865854    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3813a1-746c989f75-h9nc2"]
Nov 01 22:44:02.859874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:02.859839    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" path="/var/lib/kubelet/pods/0a0026ec-6f2b-4b00-b0d8-4fe060780adf/volumes"
Nov 01 22:44:04.476125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:04.476076    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing" containerID="cri-o://7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05" gracePeriod=25
Nov 01 22:44:04.494653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:04.494618    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:44:04.815250 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:44:04.815170    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:44:14.494439 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:14.494397    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:44:24.493996 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:44:24.493974    5845 logging.go:55] [core] [Channel #8095 SubChannel #8096]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.209:50052", ServerName: "10.134.0.209:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.209:50052: connect: connection refused"
Nov 01 22:44:24.728260 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.728233    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:44:24.789770 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.789709    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\" (UniqueName: \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\") pod \"fedec63e-7e14-4617-b95a-9e09489f982b\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") "
Nov 01 22:44:24.789919 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.789781    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\") pod \"fedec63e-7e14-4617-b95a-9e09489f982b\" (UID: \"fedec63e-7e14-4617-b95a-9e09489f982b\") "
Nov 01 22:44:24.790177 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.790153    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume" pod "fedec63e-7e14-4617-b95a-9e09489f982b" (UID: "fedec63e-7e14-4617-b95a-9e09489f982b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:24.791846 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.791816    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2" pod "fedec63e-7e14-4617-b95a-9e09489f982b" (UID: "fedec63e-7e14-4617-b95a-9e09489f982b"). InnerVolumeSpecName "kube-api-access-9n7d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:24.891367 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.891342    5845 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fedec63e-7e14-4617-b95a-9e09489f982b-plugins-config-volume\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:24.891473 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.891370    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7d2\" (UniqueName: \"kubernetes.io/projected/fedec63e-7e14-4617-b95a-9e09489f982b-kube-api-access-9n7d2\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:24.977605 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.977573    5845 generic.go:356] "Generic (PLEG): container finished" podID="fedec63e-7e14-4617-b95a-9e09489f982b" containerID="7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05" exitCode=0
Nov 01 22:44:24.977732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.977617    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"
Nov 01 22:44:24.977732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.977642    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" event={"ID":"fedec63e-7e14-4617-b95a-9e09489f982b","Type":"ContainerDied","Data":"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"}
Nov 01 22:44:24.977732 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.977698    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" event={"ID":"fedec63e-7e14-4617-b95a-9e09489f982b","Type":"ContainerDied","Data":"498105331c8561015dcdd241571a01b2a013943ce8bea32a7db954a61f1872d6"}
Nov 01 22:44:24.977973 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.977726    5845 scope.go:122] "RemoveContainer" containerID="7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"
Nov 01 22:44:24.987138 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.987116    5845 scope.go:122] "RemoveContainer" containerID="7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"
Nov 01 22:44:24.987419 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:44:24.987399    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05\": container with ID starting with 7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05 not found: ID does not exist" containerID="7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"
Nov 01 22:44:24.987474 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.987426    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05"} err="failed to get container status \"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05\": rpc error: code = NotFound desc = could not find container \"7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05\": container with ID starting with 7bd41db0eae1e2d8d4a893c84caed2ae008e5e703a73d9c1ecbd6be61d6b9e05 not found: ID does not exist"
Nov 01 22:44:24.995689 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.995667    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:44:24.999924 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:24.999894    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8"]
Nov 01 22:44:25.493662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:25.493622    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3813a1-6f6bd99cbhd6z8" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.209:50052\" within 1s: context deadline exceeded"
Nov 01 22:44:26.859917 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:26.859885    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" path="/var/lib/kubelet/pods/fedec63e-7e14-4617-b95a-9e09489f982b/volumes"
Nov 01 22:44:34.818229 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:44:34.818190    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:44:59.444197 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:59.444172    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:44:59.454115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:59.454092    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:44:59.454555 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:59.454536    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:44:59.463697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:44:59.463675    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:45:04.815477 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:45:04.815437    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:45:30.657744 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.657712    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"]
Nov 01 22:45:30.658213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658132    5845 cpu_manager.go:414] "containerMap: removing container" podUID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" containerName="maas-api"
Nov 01 22:45:30.658213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658143    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" containerName="maas-api"
Nov 01 22:45:30.658213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658158    5845 cpu_manager.go:414] "containerMap: removing container" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing"
Nov 01 22:45:30.658213 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658163    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing"
Nov 01 22:45:30.658388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658244    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0026ec-6f2b-4b00-b0d8-4fe060780adf" containerName="maas-api"
Nov 01 22:45:30.658388 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.658252    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="fedec63e-7e14-4617-b95a-9e09489f982b" containerName="payload-processing"
Nov 01 22:45:30.662684 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.662662    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.678807 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.678759    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"]
Nov 01 22:45:30.821901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.821867    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256642bb-101e-4db6-bb35-ae0caaca75ed-openshift-service-ca\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.822113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.821923    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-maas-api-tls\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.822113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.822071    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7zr\" (UniqueName: \"kubernetes.io/projected/256642bb-101e-4db6-bb35-ae0caaca75ed-kube-api-access-ns7zr\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.822113 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.822108    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-metrics-certs\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.923048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.922960    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256642bb-101e-4db6-bb35-ae0caaca75ed-openshift-service-ca\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.923048 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.923009    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-maas-api-tls\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.923262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.923065    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7zr\" (UniqueName: \"kubernetes.io/projected/256642bb-101e-4db6-bb35-ae0caaca75ed-kube-api-access-ns7zr\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.923262 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.923082    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-metrics-certs\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.923685 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.923664    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/256642bb-101e-4db6-bb35-ae0caaca75ed-openshift-service-ca\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.925624 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.925603    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-metrics-certs\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.925722 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.925701    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/256642bb-101e-4db6-bb35-ae0caaca75ed-maas-api-tls\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.935371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.935344    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7zr\" (UniqueName: \"kubernetes.io/projected/256642bb-101e-4db6-bb35-ae0caaca75ed-kube-api-access-ns7zr\") pod \"maas-api-598d6b8b55-vmzqm\" (UID: \"256642bb-101e-4db6-bb35-ae0caaca75ed\") " pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:30.989342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:30.989287    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:31.271548 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:31.271522    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"]
Nov 01 22:45:31.274336 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:45:31.274279    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256642bb_101e_4db6_bb35_ae0caaca75ed.slice/crio-cbc01f9efb01860c055305975a377f1ce96268e70779cab97b572ec2dd439a18 WatchSource:0}: Error finding container cbc01f9efb01860c055305975a377f1ce96268e70779cab97b572ec2dd439a18: Status 404 returned error can't find the container with id cbc01f9efb01860c055305975a377f1ce96268e70779cab97b572ec2dd439a18
Nov 01 22:45:31.275697 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:31.275673    5845 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:45:31.419518 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:31.419476    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm" event={"ID":"256642bb-101e-4db6-bb35-ae0caaca75ed","Type":"ContainerStarted","Data":"cbc01f9efb01860c055305975a377f1ce96268e70779cab97b572ec2dd439a18"}
Nov 01 22:45:32.426111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:32.426073    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm" event={"ID":"256642bb-101e-4db6-bb35-ae0caaca75ed","Type":"ContainerStarted","Data":"e1e4b5138bc7df64bcf3b8d0720d74987f92991aac55465e24dbe0fa106a007e"}
Nov 01 22:45:32.426842 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:32.426131    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:32.446835 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:32.446770    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm" podStartSLOduration=2.103197475 podStartE2EDuration="2.446751213s" totalImagesPullingTime="343.553738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:32.445979951 +0000 UTC m=+3635.702159714" watchObservedRunningTime="2026-11-01 22:45:32.446751213 +0000 UTC m=+3635.702930975"
Nov 01 22:45:34.815019 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:45:34.814984    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:45:37.786756 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.786718    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:45:37.790788 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.790768    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.798753 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.798731    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eaf797bf-openshift-default-dockercfg-5f6gg\""
Nov 01 22:45:37.798922 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.798900    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.798993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.798936    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.798994    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799020    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799053 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799046    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4p5\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799128    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799163    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799194    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.799283 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.799217    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.805691 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.805666    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:45:37.900494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900457    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900494 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900499    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900521    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900546    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900578    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4p5\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900645    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.900810 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900669    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900978    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.901091    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.901103    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.900789    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.901162    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.901539    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.901679 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.901586    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.903032 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.903010    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.903166 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.903148    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.913124 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.913100    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4p5\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:37.913404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:37.913387    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\") pod \"e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") " pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:38.104348 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.104254    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:38.386628 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.386604    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:45:38.388928 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:45:38.388896    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3179e085_c605_41ff_bc09_361743625904.slice/crio-22cb4bb24d21623ecd82cbea370dd2d0b09443f74e4938d0060c23620b236439 WatchSource:0}: Error finding container 22cb4bb24d21623ecd82cbea370dd2d0b09443f74e4938d0060c23620b236439: Status 404 returned error can't find the container with id 22cb4bb24d21623ecd82cbea370dd2d0b09443f74e4938d0060c23620b236439
Nov 01 22:45:38.391115 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.391088    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:38.391185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.391146    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:38.391185 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.391175    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:38.437170 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.437151    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-598d6b8b55-vmzqm"
Nov 01 22:45:38.457657 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.457608    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerStarted","Data":"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"}
Nov 01 22:45:38.457812 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.457691    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerStarted","Data":"22cb4bb24d21623ecd82cbea370dd2d0b09443f74e4938d0060c23620b236439"}
Nov 01 22:45:38.483116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:38.483059    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podStartSLOduration=1.483041829 podStartE2EDuration="1.483041829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:38.481347206 +0000 UTC m=+3641.737526972" watchObservedRunningTime="2026-11-01 22:45:38.483041829 +0000 UTC m=+3641.739221591"
Nov 01 22:45:39.105410 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:39.105368    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:39.110817 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:39.110788    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:39.462492 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:39.462413    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:39.463398 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:39.463376    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:46.627687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:46.627650    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 22:45:46.628146 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:46.627864    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api" containerID="cri-o://e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de" gracePeriod=30
Nov 01 22:45:47.929178 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:47.929152    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 22:45:48.014658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.014568    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\") pod \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") "
Nov 01 22:45:48.014658 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.014629    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\") pod \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") "
Nov 01 22:45:48.014976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.014678    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\" (UniqueName: \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\") pod \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") "
Nov 01 22:45:48.014976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.014743    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\") pod \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\" (UID: \"889f0c02-0248-4f0e-afc9-f3f7730f3aac\") "
Nov 01 22:45:48.015089 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.014991    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca" pod "889f0c02-0248-4f0e-afc9-f3f7730f3aac" (UID: "889f0c02-0248-4f0e-afc9-f3f7730f3aac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:48.016837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.016813    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs" pod "889f0c02-0248-4f0e-afc9-f3f7730f3aac" (UID: "889f0c02-0248-4f0e-afc9-f3f7730f3aac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:48.016944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.016931    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq" pod "889f0c02-0248-4f0e-afc9-f3f7730f3aac" (UID: "889f0c02-0248-4f0e-afc9-f3f7730f3aac"). InnerVolumeSpecName "kube-api-access-b74jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:48.016944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.016933    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls" pod "889f0c02-0248-4f0e-afc9-f3f7730f3aac" (UID: "889f0c02-0248-4f0e-afc9-f3f7730f3aac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:48.115867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.115831    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/889f0c02-0248-4f0e-afc9-f3f7730f3aac-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:48.115867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.115860    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:48.115867 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.115870    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74jq\" (UniqueName: \"kubernetes.io/projected/889f0c02-0248-4f0e-afc9-f3f7730f3aac-kube-api-access-b74jq\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:48.116107 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.115878    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/889f0c02-0248-4f0e-afc9-f3f7730f3aac-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:48.509312 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.509256    5845 generic.go:356] "Generic (PLEG): container finished" podID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerID="e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de" exitCode=0
Nov 01 22:45:48.509471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.509339    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"
Nov 01 22:45:48.509471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.509336    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" event={"ID":"889f0c02-0248-4f0e-afc9-f3f7730f3aac","Type":"ContainerDied","Data":"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"}
Nov 01 22:45:48.509471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.509421    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-976bc5795-qwgk4" event={"ID":"889f0c02-0248-4f0e-afc9-f3f7730f3aac","Type":"ContainerDied","Data":"8fd827a7548c9c76eb5e261d093b9e9b0a9424e89cec6fb9f9d67f86db228c20"}
Nov 01 22:45:48.509471 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.509445    5845 scope.go:122] "RemoveContainer" containerID="e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"
Nov 01 22:45:48.523345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.523320    5845 scope.go:122] "RemoveContainer" containerID="e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"
Nov 01 22:45:48.523657 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:45:48.523632    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de\": container with ID starting with e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de not found: ID does not exist" containerID="e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"
Nov 01 22:45:48.523742 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.523666    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de"} err="failed to get container status \"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de\": rpc error: code = NotFound desc = could not find container \"e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de\": container with ID starting with e28eef6c1321cb3db4aeb1539575021659dfa7670e6bef8a80864c0eed32b4de not found: ID does not exist"
Nov 01 22:45:48.535006 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.534975    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 22:45:48.538961 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.538935    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-976bc5795-qwgk4"]
Nov 01 22:45:48.876269 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:48.876186    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" path="/var/lib/kubelet/pods/889f0c02-0248-4f0e-afc9-f3f7730f3aac/volumes"
Nov 01 22:45:53.536601 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.536568    5845 generic.go:356] "Generic (PLEG): container finished" podID="3179e085-c605-41ff-bc09-361743625904" containerID="1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793" exitCode=0
Nov 01 22:45:53.537039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.536644    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerDied","Data":"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"}
Nov 01 22:45:53.537039 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.537029    5845 scope.go:122] "RemoveContainer" containerID="1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"
Nov 01 22:45:53.538252 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.538222    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:53.538386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.538285    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:53.538386 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:53.538335    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:54.466321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:54.465403    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:54.543396 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:54.543361    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerStarted","Data":"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"}
Nov 01 22:45:55.104993 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:55.104945    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:45:55.105855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:55.105829    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Nov 01 22:45:55.105976 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:55.105885    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Nov 01 22:45:56.105616 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:56.105576    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Nov 01 22:45:56.106033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:56.105657    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Nov 01 22:45:57.105007 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:57.104974    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Nov 01 22:45:57.105189 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:57.105028    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Nov 01 22:45:58.105035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:58.105004    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Nov 01 22:45:58.105462 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:58.105056    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Nov 01 22:45:59.105747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:59.105713    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body=
Nov 01 22:45:59.106134 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:45:59.105767    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused"
Nov 01 22:46:00.109414 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:00.109380    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:46:00.109840 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:00.109632    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:46:00.110345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:00.110326    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:46:04.819317 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:46:04.819263    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:46:12.092707 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:12.092659    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:46:12.093158 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:12.093017    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" containerID="cri-o://3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad" gracePeriod=30
Nov 01 22:46:13.731065 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.731027    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:46:13.731511 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.731495    5845 cpu_manager.go:414] "containerMap: removing container" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api"
Nov 01 22:46:13.731557 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.731512    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api"
Nov 01 22:46:13.731599 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.731577    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="889f0c02-0248-4f0e-afc9-f3f7730f3aac" containerName="maas-api"
Nov 01 22:46:13.734898 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.734874    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.739712 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.739677    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-1fef08-openshift-default-dockercfg-9wcxk\""
Nov 01 22:46:13.747585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.747562    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:46:13.882813 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.882779    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcw4v\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.882965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.882859    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.882965 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.882921    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883055 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.882971    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.883051    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.883076    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.883092    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883196 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.883107    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.883248 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.883228    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984036 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.983948    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcw4v\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984090    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984130    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984174    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984198 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984192    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984217    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984238    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984263    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984504 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984318    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984592    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984893 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984859    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.984985 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.984903    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.985153 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.985133    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.985226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.985183    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.986677 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.986651    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.986776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.986764    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.996725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.996698    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:13.996851 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:13.996830    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcw4v\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\") pod \"e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") " pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:14.048763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.048737    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:14.327549 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.327514    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:46:14.330102 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:46:14.330067    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed84219_6034_4690_9b92_e59eb7661e67.slice/crio-c61164857fbaccc1131e05b95bf1b7fafe988f751901949ef58043a9e5adbb71 WatchSource:0}: Error finding container c61164857fbaccc1131e05b95bf1b7fafe988f751901949ef58043a9e5adbb71: Status 404 returned error can't find the container with id c61164857fbaccc1131e05b95bf1b7fafe988f751901949ef58043a9e5adbb71
Nov 01 22:46:14.332116 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.332087    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:14.332210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.332149    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:14.332210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.332176    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:14.665948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.665866    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerStarted","Data":"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"}
Nov 01 22:46:14.665948 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.665929    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerStarted","Data":"c61164857fbaccc1131e05b95bf1b7fafe988f751901949ef58043a9e5adbb71"}
Nov 01 22:46:14.688671 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:14.688615    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podStartSLOduration=1.688596992 podStartE2EDuration="1.688596992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:14.687765883 +0000 UTC m=+3677.943945647" watchObservedRunningTime="2026-11-01 22:46:14.688596992 +0000 UTC m=+3677.944776757"
Nov 01 22:46:15.049632 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.049599    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:15.054538 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.054511    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:15.112724 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.112688    5845 patch_prober.go:29] interesting pod/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:46:15.112874 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.112744    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:46:15.671725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.671694    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:15.672670 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:15.672647    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:17.359464 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.359404    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:46:17.521725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521683    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.521940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521818    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.521940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521867    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.521940 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521896    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521946    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.521985    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522051    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522101    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522140    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\") pod \"3179e085-c605-41ff-bc09-361743625904\" (UID: \"3179e085-c605-41ff-bc09-361743625904\") "
Nov 01 22:46:17.522345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522186    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:46:17.522345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522202    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:46:17.522521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522348    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:46:17.522521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522452    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:46:17.522521 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522386    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:46:17.522682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522591    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.522682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522607    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.522682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522615    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.522682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522625    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3179e085-c605-41ff-bc09-361743625904-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.522682 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.522633    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.524551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.524523    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "kube-api-access-bh4p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:46:17.524551 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.524522    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:46:17.524715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.524683    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:46:17.524715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.524706    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token" pod "3179e085-c605-41ff-bc09-361743625904" (UID: "3179e085-c605-41ff-bc09-361743625904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:46:17.624012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.623914    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh4p5\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-kube-api-access-bh4p5\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.624012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.623949    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3179e085-c605-41ff-bc09-361743625904-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.624012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.623965    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3179e085-c605-41ff-bc09-361743625904-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.624012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.623977    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3179e085-c605-41ff-bc09-361743625904-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:46:17.682577 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.682547    5845 generic.go:356] "Generic (PLEG): container finished" podID="3179e085-c605-41ff-bc09-361743625904" containerID="3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad" exitCode=0
Nov 01 22:46:17.682761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.682597    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"
Nov 01 22:46:17.682761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.682635    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerDied","Data":"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"}
Nov 01 22:46:17.682761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.682700    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn" event={"ID":"3179e085-c605-41ff-bc09-361743625904","Type":"ContainerDied","Data":"22cb4bb24d21623ecd82cbea370dd2d0b09443f74e4938d0060c23620b236439"}
Nov 01 22:46:17.682761 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.682729    5845 scope.go:122] "RemoveContainer" containerID="3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"
Nov 01 22:46:17.692267 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.692251    5845 scope.go:122] "RemoveContainer" containerID="1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"
Nov 01 22:46:17.700380 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.700363    5845 scope.go:122] "RemoveContainer" containerID="3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"
Nov 01 22:46:17.700651 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:46:17.700632    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad\": container with ID starting with 3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad not found: ID does not exist" containerID="3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"
Nov 01 22:46:17.700702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.700661    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad"} err="failed to get container status \"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad\": rpc error: code = NotFound desc = could not find container \"3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad\": container with ID starting with 3a50035e9b392a4dfc45d89ba4cd13af012fa3dc2164feaa5f46f273ab51c2ad not found: ID does not exist"
Nov 01 22:46:17.700702 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.700678    5845 scope.go:122] "RemoveContainer" containerID="1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"
Nov 01 22:46:17.700880 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:46:17.700861    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793\": container with ID starting with 1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793 not found: ID does not exist" containerID="1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"
Nov 01 22:46:17.700923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.700889    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793"} err="failed to get container status \"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793\": rpc error: code = NotFound desc = could not find container \"1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793\": container with ID starting with 1d52b7a35a4a969fdd2596ac5008bb67a787a955108d9bc380c105f3337a7793 not found: ID does not exist"
Nov 01 22:46:17.707491 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.707467    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:46:17.711723 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:17.711700    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eaf797bf-openshift-default-75d787674c-rnwcn"]
Nov 01 22:46:18.079228 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.079196    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:46:18.079747 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.079727    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:46:18.079845 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.079751    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:46:18.079951 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.079937    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:46:18.090182 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.090157    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.094427 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.094395    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:46:18.098585 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.098553    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-1fef08-serving-cert\""
Nov 01 22:46:18.229634 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.229602    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.229799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.229649    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.229799 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.229675    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njct8\" (UniqueName: \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.229910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.229843    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.331075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.330979    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.331075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.331028    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njct8\" (UniqueName: \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.331075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.331057    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.331345 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.331145    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.331824 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.331799    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.333567 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.333547    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.333725 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.333702    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.343374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.343349    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njct8\" (UniqueName: \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\") pod \"maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.410226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.410188    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:18.687438 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.687353    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:46:18.691929 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:46:18.691901    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd266ed69_f621_4c53_94c1_b82c50d1e270.slice/crio-538b622554a5f0e3a991167eb32c31bc3cc4cb9807ed18d11f456687deeda3bf WatchSource:0}: Error finding container 538b622554a5f0e3a991167eb32c31bc3cc4cb9807ed18d11f456687deeda3bf: Status 404 returned error can't find the container with id 538b622554a5f0e3a991167eb32c31bc3cc4cb9807ed18d11f456687deeda3bf
Nov 01 22:46:18.859564 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:18.859533    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3179e085-c605-41ff-bc09-361743625904" path="/var/lib/kubelet/pods/3179e085-c605-41ff-bc09-361743625904/volumes"
Nov 01 22:46:19.700814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:19.698894    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" event={"ID":"d266ed69-f621-4c53-94c1-b82c50d1e270","Type":"ContainerStarted","Data":"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"}
Nov 01 22:46:19.700814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:19.699038    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" event={"ID":"d266ed69-f621-4c53-94c1-b82c50d1e270","Type":"ContainerStarted","Data":"538b622554a5f0e3a991167eb32c31bc3cc4cb9807ed18d11f456687deeda3bf"}
Nov 01 22:46:19.700814 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:19.699353    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:19.721926 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:19.721876    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" podStartSLOduration=1.302143262 podStartE2EDuration="1.721864779s" totalImagesPullingTime="419.721517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:19.720957757 +0000 UTC m=+3682.977137519" watchObservedRunningTime="2026-11-01 22:46:19.721864779 +0000 UTC m=+3682.978044536"
Nov 01 22:46:25.709958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:25.709919    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:46:30.673487 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.673450    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:30.673910 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.673504    5845 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:30.758183 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.758150    5845 generic.go:356] "Generic (PLEG): container finished" podID="7ed84219-6034-4690-9b92-e59eb7661e67" containerID="a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c" exitCode=0
Nov 01 22:46:30.758364 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.758219    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerDied","Data":"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"}
Nov 01 22:46:30.758617 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.758603    5845 scope.go:122] "RemoveContainer" containerID="a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"
Nov 01 22:46:30.760075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.760043    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:30.760223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.760128    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:30.760223 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:30.760176    5845 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:31.773894 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:31.773856    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerStarted","Data":"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"}
Nov 01 22:46:32.062381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:32.057698    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:32.062381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:32.057823    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:32.062381 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:32.057880    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:33.063321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:33.053503    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:33.063321 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:33.053578    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:34.055736 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:34.055696    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:34.056055 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:34.056025    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:34.818184 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:46:34.818102    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:46:35.066320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:35.065697    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:35.066320 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:35.065762    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:36.052714 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:36.052676    5845 patch_prober.go:29] interesting pod/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Nov 01 22:46:36.053342 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:36.053281    5845 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Nov 01 22:46:37.053049 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:37.053019    5845 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:37.053516 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:37.053291    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:46:37.054062 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:46:37.054042    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:47:04.815576 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:47:04.815533    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:47:34.822940 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:47:34.822896    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:48:04.815805 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:48:04.815764    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:48:07.492762 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.492724    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:48:07.493347 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.492944    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" podUID="d266ed69-f621-4c53-94c1-b82c50d1e270" containerName="maas-api" containerID="cri-o://5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0" gracePeriod=30
Nov 01 22:48:07.740210 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.740187    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:48:07.868290 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.868198    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\") pod \"d266ed69-f621-4c53-94c1-b82c50d1e270\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") "
Nov 01 22:48:07.868290 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.868267    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\") pod \"d266ed69-f621-4c53-94c1-b82c50d1e270\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") "
Nov 01 22:48:07.868510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.868293    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\") pod \"d266ed69-f621-4c53-94c1-b82c50d1e270\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") "
Nov 01 22:48:07.868510 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.868354    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\" (UniqueName: \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\") pod \"d266ed69-f621-4c53-94c1-b82c50d1e270\" (UID: \"d266ed69-f621-4c53-94c1-b82c50d1e270\") "
Nov 01 22:48:07.868652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.868625    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca" pod "d266ed69-f621-4c53-94c1-b82c50d1e270" (UID: "d266ed69-f621-4c53-94c1-b82c50d1e270"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:07.870418 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.870390    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8" pod "d266ed69-f621-4c53-94c1-b82c50d1e270" (UID: "d266ed69-f621-4c53-94c1-b82c50d1e270"). InnerVolumeSpecName "kube-api-access-njct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:07.870506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.870480    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs" pod "d266ed69-f621-4c53-94c1-b82c50d1e270" (UID: "d266ed69-f621-4c53-94c1-b82c50d1e270"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:07.870506 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.870492    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls" pod "d266ed69-f621-4c53-94c1-b82c50d1e270" (UID: "d266ed69-f621-4c53-94c1-b82c50d1e270"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:07.969675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.969634    5845 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d266ed69-f621-4c53-94c1-b82c50d1e270-openshift-service-ca\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:07.969675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.969670    5845 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-metrics-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:07.969675 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.969684    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njct8\" (UniqueName: \"kubernetes.io/projected/d266ed69-f621-4c53-94c1-b82c50d1e270-kube-api-access-njct8\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:07.969916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:07.969699    5845 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d266ed69-f621-4c53-94c1-b82c50d1e270-maas-api-tls\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:08.268045 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.268009    5845 generic.go:356] "Generic (PLEG): container finished" podID="d266ed69-f621-4c53-94c1-b82c50d1e270" containerID="5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0" exitCode=0
Nov 01 22:48:08.268237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.268061    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"
Nov 01 22:48:08.268237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.268100    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" event={"ID":"d266ed69-f621-4c53-94c1-b82c50d1e270","Type":"ContainerDied","Data":"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"}
Nov 01 22:48:08.268237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.268160    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf" event={"ID":"d266ed69-f621-4c53-94c1-b82c50d1e270","Type":"ContainerDied","Data":"538b622554a5f0e3a991167eb32c31bc3cc4cb9807ed18d11f456687deeda3bf"}
Nov 01 22:48:08.268237 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.268189    5845 scope.go:122] "RemoveContainer" containerID="5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"
Nov 01 22:48:08.279980 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.279959    5845 scope.go:122] "RemoveContainer" containerID="5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"
Nov 01 22:48:08.280326 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:48:08.280267    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0\": container with ID starting with 5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0 not found: ID does not exist" containerID="5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"
Nov 01 22:48:08.280425 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.280384    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0"} err="failed to get container status \"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0\": rpc error: code = NotFound desc = could not find container \"5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0\": container with ID starting with 5a1da022a460205553117a0792d5e280340c5c9b6cca9837ea2cd4892583b8b0 not found: ID does not exist"
Nov 01 22:48:08.292921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.292898    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:48:08.296819 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.296797    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-1fef08-6bb8cc7c64-9w4pf"]
Nov 01 22:48:08.860449 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:08.860414    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d266ed69-f621-4c53-94c1-b82c50d1e270" path="/var/lib/kubelet/pods/d266ed69-f621-4c53-94c1-b82c50d1e270/volumes"
Nov 01 22:48:14.377779 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:14.377733    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:48:14.378308 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:14.378081    5845 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy" containerID="cri-o://2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3" gracePeriod=30
Nov 01 22:48:19.640958 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.640931    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:48:19.816608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816574    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.816781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816624    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.816781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816681    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.816781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816712    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.816781 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816758    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.817012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816810    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.817012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816842    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.817012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816878    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.817012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816938    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:19.817012 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.816963    5845 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\") pod \"7ed84219-6034-4690-9b92-e59eb7661e67\" (UID: \"7ed84219-6034-4690-9b92-e59eb7661e67\") "
Nov 01 22:48:19.817323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817171    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:19.817323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817259    5845 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-credential-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.817323 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817275    5845 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-data\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.817545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817375    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:19.817545 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817388    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:19.817700 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.817603    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:19.819034 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.818998    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:48:19.819161 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.819037    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:19.819537 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.819503    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "kube-api-access-hcw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:19.819715 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.819689    5845 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo" pod "7ed84219-6034-4690-9b92-e59eb7661e67" (UID: "7ed84219-6034-4690-9b92-e59eb7661e67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:48:19.917952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.917915    5845 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-istio-token\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.917952 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.917946    5845 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-socket\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.918150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.917960    5845 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-workload-certs\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.918150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.917975    5845 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed84219-6034-4690-9b92-e59eb7661e67-istio-podinfo\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.918150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.917989    5845 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcw4v\" (UniqueName: \"kubernetes.io/projected/7ed84219-6034-4690-9b92-e59eb7661e67-kube-api-access-hcw4v\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.918150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.918002    5845 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed84219-6034-4690-9b92-e59eb7661e67-istiod-ca-cert\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:19.918150 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:19.918015    5845 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed84219-6034-4690-9b92-e59eb7661e67-istio-envoy\") on node \"ip-10-0-135-146.us-east-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:20.334461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.334427    5845 generic.go:356] "Generic (PLEG): container finished" podID="7ed84219-6034-4690-9b92-e59eb7661e67" containerID="2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3" exitCode=0
Nov 01 22:48:20.334654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.334489    5845 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"
Nov 01 22:48:20.334654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.334498    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerDied","Data":"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"}
Nov 01 22:48:20.334654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.334601    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n" event={"ID":"7ed84219-6034-4690-9b92-e59eb7661e67","Type":"ContainerDied","Data":"c61164857fbaccc1131e05b95bf1b7fafe988f751901949ef58043a9e5adbb71"}
Nov 01 22:48:20.334654 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.334621    5845 scope.go:122] "RemoveContainer" containerID="2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"
Nov 01 22:48:20.345120 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.345101    5845 scope.go:122] "RemoveContainer" containerID="a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"
Nov 01 22:48:20.353125 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.353103    5845 scope.go:122] "RemoveContainer" containerID="2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"
Nov 01 22:48:20.353392 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:48:20.353373    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3\": container with ID starting with 2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3 not found: ID does not exist" containerID="2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"
Nov 01 22:48:20.353453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.353401    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3"} err="failed to get container status \"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3\": rpc error: code = NotFound desc = could not find container \"2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3\": container with ID starting with 2bb717a9b2f4bcc88690916f341dccdf4ee12f6c3cf1657c59b8f57777385bc3 not found: ID does not exist"
Nov 01 22:48:20.353453 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.353416    5845 scope.go:122] "RemoveContainer" containerID="a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"
Nov 01 22:48:20.353666 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:48:20.353648    5845 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c\": container with ID starting with a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c not found: ID does not exist" containerID="a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"
Nov 01 22:48:20.353708 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.353673    5845 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c"} err="failed to get container status \"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c\": rpc error: code = NotFound desc = could not find container \"a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c\": container with ID starting with a5ef885198f92f61b8f48b9207ab08cd9194d49e1ad3539bc7e4bf50c14cdc9c not found: ID does not exist"
Nov 01 22:48:20.358218 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.358196    5845 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:48:20.362111 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.362091    5845 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1fef08-openshift-default-86b4fb9d45-7jv5n"]
Nov 01 22:48:20.859372 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:48:20.859339    5845 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" path="/var/lib/kubelet/pods/7ed84219-6034-4690-9b92-e59eb7661e67/volumes"
Nov 01 22:48:34.815061 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:48:34.815024    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:49:03.533763 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:03.533731    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697bf54cfb-r86kt_12e40591-1504-4ecd-a486-c6607a5c0225/authorino/0.log"
Nov 01 22:49:04.842756 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:49:04.842714    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:49:13.916114 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:13.916086    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dz5ct_d1f29aa3-d6a1-4153-b759-0b130813ebf0/manager/1.log"
Nov 01 22:49:15.295096 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:15.295063    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dz5ct_d1f29aa3-d6a1-4153-b759-0b130813ebf0/manager/0.log"
Nov 01 22:49:16.016841 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:16.016805    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-598d6b8b55-vmzqm_256642bb-101e-4db6-bb35-ae0caaca75ed/maas-api/0.log"
Nov 01 22:49:18.630944 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:18.630907    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697bf54cfb-r86kt_12e40591-1504-4ecd-a486-c6607a5c0225/authorino/0.log"
Nov 01 22:49:20.961123 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:20.961088    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-kdssh_ed7e30f6-adec-4f93-a098-2ab4f61f937d/kube-auth-proxy/0.log"
Nov 01 22:49:21.477313 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:21.477270    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fp2b7_72fb2f07-f693-45bc-90e5-f622fdef72e5/istio-proxy/0.log"
Nov 01 22:49:22.192234 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:22.192203    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-mrlcb_73a30e41-ed57-4bff-92a2-38160219bfb5/istio-proxy/0.log"
Nov 01 22:49:22.345035 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:22.345006    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-prd6f_b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec/istio-proxy/0.log"
Nov 01 22:49:24.606393 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:24.606360    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fszvp_2c63df49-6605-4914-9756-a8dcb41d4a15/main/0.log"
Nov 01 22:49:24.775541 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:24.775503    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm_0a2d2561-28ba-4f2e-bd41-88b863f73b11/main/0.log"
Nov 01 22:49:24.939637 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:24.939525    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cdk5r_b27c16ff-fa1f-4815-902c-8a05f5070f07/main/0.log"
Nov 01 22:49:25.269063 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:25.269026    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m_83f9f654-a050-4cdc-bc76-d2ea43698bdd/main/0.log"
Nov 01 22:49:26.116245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:26.116210    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697bf54cfb-r86kt_12e40591-1504-4ecd-a486-c6607a5c0225/authorino/0.log"
Nov 01 22:49:34.815189 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:49:34.815154    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:49:35.035070 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:35.035037    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dz5ct_d1f29aa3-d6a1-4153-b759-0b130813ebf0/manager/1.log"
Nov 01 22:49:36.708662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:36.708628    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-598d6b8b55-vmzqm_256642bb-101e-4db6-bb35-ae0caaca75ed/maas-api/0.log"
Nov 01 22:49:38.972578 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:38.972486    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697bf54cfb-r86kt_12e40591-1504-4ecd-a486-c6607a5c0225/authorino/0.log"
Nov 01 22:49:41.120806 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:41.120773    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-kdssh_ed7e30f6-adec-4f93-a098-2ab4f61f937d/kube-auth-proxy/0.log"
Nov 01 22:49:41.622130 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:41.622096    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fp2b7_72fb2f07-f693-45bc-90e5-f622fdef72e5/istio-proxy/0.log"
Nov 01 22:49:42.330847 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:42.330812    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-mrlcb_73a30e41-ed57-4bff-92a2-38160219bfb5/istio-proxy/0.log"
Nov 01 22:49:42.502243 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:42.502213    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-prd6f_b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec/istio-proxy/0.log"
Nov 01 22:49:44.560095 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:44.560046    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fszvp_2c63df49-6605-4914-9756-a8dcb41d4a15/main/0.log"
Nov 01 22:49:44.728359 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:44.728327    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b886kgm_0a2d2561-28ba-4f2e-bd41-88b863f73b11/main/0.log"
Nov 01 22:49:44.894766 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:44.894685    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cdk5r_b27c16ff-fa1f-4815-902c-8a05f5070f07/main/0.log"
Nov 01 22:49:45.232432 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:45.232400    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6rz6m_83f9f654-a050-4cdc-bc76-d2ea43698bdd/main/0.log"
Nov 01 22:49:59.488669 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:59.488639    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:49:59.493163 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:59.493142    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:49:59.497226 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:59.497209    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-98dc2_98dd454d-3bb0-48d9-b6ed-d907da241372/csi-driver/0.log"
Nov 01 22:49:59.501186 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:49:59.501167    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw7wh_2ed0239d-76c2-4d3b-b168-560c022e4e3a/ovn-acl-logging/0.log"
Nov 01 22:50:04.814025 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:50:04.813986    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:50:34.816414 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:50:34.816377    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:51:04.815749 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:51:04.815710    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:51:08.374759 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:08.374671    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f4s8z_c2851cfd-c9b5-40db-8ddb-93ad5ba9fb61/global-pull-secret-syncer/0.log"
Nov 01 22:51:08.733148 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:08.733105    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9wtgk_dac098ba-33a8-4a99-812f-ae5561684d7c/konnectivity-agent/0.log"
Nov 01 22:51:08.799916 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:08.799878    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-146.us-east-2.compute.internal_ec31c2ba0ee2bf174eddd7fce91e152b/haproxy/0.log"
Nov 01 22:51:16.397855 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:16.397818    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697bf54cfb-r86kt_12e40591-1504-4ecd-a486-c6607a5c0225/authorino/0.log"
Nov 01 22:51:17.715870 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.715826    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/alertmanager/0.log"
Nov 01 22:51:17.740775 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.740745    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/config-reloader/0.log"
Nov 01 22:51:17.764056 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.764031    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/kube-rbac-proxy-web/0.log"
Nov 01 22:51:17.786875 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.786853    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/kube-rbac-proxy/0.log"
Nov 01 22:51:17.811416 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.811388    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/kube-rbac-proxy-metric/0.log"
Nov 01 22:51:17.847687 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.847652    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/prom-label-proxy/0.log"
Nov 01 22:51:17.873421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.873403    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_051fd871-015f-445a-8314-b557c25f0443/init-config-reloader/0.log"
Nov 01 22:51:17.967061 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.967040    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-97sfn_5364a66c-4dcd-4819-a8a0-eb48a1b57f33/kube-state-metrics/0.log"
Nov 01 22:51:17.982979 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.982958    5845 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"]
Nov 01 22:51:17.983374 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983361    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983375    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983384    5845 cpu_manager.go:414] "containerMap: removing container" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983390    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983403    5845 cpu_manager.go:414] "containerMap: removing container" podUID="d266ed69-f621-4c53-94c1-b82c50d1e270" containerName="maas-api"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983407    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d266ed69-f621-4c53-94c1-b82c50d1e270" containerName="maas-api"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983414    5845 cpu_manager.go:414] "containerMap: removing container" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:51:17.983426 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983420    5845 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:51:17.983662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983503    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983512    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed84219-6034-4690-9b92-e59eb7661e67" containerName="istio-proxy"
Nov 01 22:51:17.983662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983519    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="d266ed69-f621-4c53-94c1-b82c50d1e270" containerName="maas-api"
Nov 01 22:51:17.983662 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.983525    5845 memory_manager.go:365] "RemoveStaleState removing state" podUID="3179e085-c605-41ff-bc09-361743625904" containerName="istio-proxy"
Nov 01 22:51:17.986608 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.986594    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:17.990576 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.990559    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-97sfn_5364a66c-4dcd-4819-a8a0-eb48a1b57f33/kube-rbac-proxy-main/0.log"
Nov 01 22:51:17.998774 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:17.998754    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"]
Nov 01 22:51:18.019340 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.019320    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-97sfn_5364a66c-4dcd-4819-a8a0-eb48a1b57f33/kube-rbac-proxy-self/0.log"
Nov 01 22:51:18.045934 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.045916    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f99686fb5-2wbs5_cfeec369-c249-494b-b2a2-8c0753b3d17e/metrics-server/1.log"
Nov 01 22:51:18.047064 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.047022    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f99686fb5-2wbs5_cfeec369-c249-494b-b2a2-8c0753b3d17e/metrics-server/0.log"
Nov 01 22:51:18.065358 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.065335    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-proc\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.065461 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.065378    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-podres\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.065526 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.065504    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-lib-modules\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.065630 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.065610    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-sys\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.065667 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.065653    5845 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csqj\" (UniqueName: \"kubernetes.io/projected/1c6e83f1-3ddd-443a-887f-606c14dd5c72-kube-api-access-6csqj\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.069287 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.069265    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-kwlgq_cefe7ab7-6f8a-4895-af7f-747da1036866/monitoring-plugin/0.log"
Nov 01 22:51:18.082776 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.082759    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"openshift-service-ca.crt\""
Nov 01 22:51:18.082869 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.082844    5845 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gx6ks\"/\"kube-root-ca.crt\""
Nov 01 22:51:18.082911 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.082866    5845 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gx6ks\"/\"default-dockercfg-8xcsn\""
Nov 01 22:51:18.166404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166368    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-proc\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166404 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166408    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-podres\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166443    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-lib-modules\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166480    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-sys\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166490    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-proc\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166505    5845 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csqj\" (UniqueName: \"kubernetes.io/projected/1c6e83f1-3ddd-443a-887f-606c14dd5c72-kube-api-access-6csqj\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166553    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-sys\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166567    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-podres\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.166652 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.166592    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c6e83f1-3ddd-443a-887f-606c14dd5c72-lib-modules\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.179164 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.179134    5845 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csqj\" (UniqueName: \"kubernetes.io/projected/1c6e83f1-3ddd-443a-887f-606c14dd5c72-kube-api-access-6csqj\") pod \"perf-node-gather-daemonset-tkzkm\" (UID: \"1c6e83f1-3ddd-443a-887f-606c14dd5c72\") " pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.288735 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.288651    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmnzz_13c6f668-3618-4d04-9942-eab2dd9e6224/node-exporter/0.log"
Nov 01 22:51:18.301022 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.300994    5845 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:18.310128 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.310096    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmnzz_13c6f668-3618-4d04-9942-eab2dd9e6224/kube-rbac-proxy/0.log"
Nov 01 22:51:18.334370 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.334345    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hmnzz_13c6f668-3618-4d04-9942-eab2dd9e6224/init-textfile/0.log"
Nov 01 22:51:18.368051 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.367973    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8vxv9_2ba32e6c-796e-49bd-8042-db4de0b9e86b/kube-rbac-proxy-main/0.log"
Nov 01 22:51:18.393147 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.393121    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8vxv9_2ba32e6c-796e-49bd-8042-db4de0b9e86b/kube-rbac-proxy-self/0.log"
Nov 01 22:51:18.421411 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.421386    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8vxv9_2ba32e6c-796e-49bd-8042-db4de0b9e86b/openshift-state-metrics/0.log"
Nov 01 22:51:18.460923 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.460898    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/prometheus/0.log"
Nov 01 22:51:18.483011 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.482985    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/config-reloader/0.log"
Nov 01 22:51:18.506536 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.506512    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/thanos-sidecar/0.log"
Nov 01 22:51:18.530082 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.530061    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/kube-rbac-proxy-web/0.log"
Nov 01 22:51:18.556167 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.556110    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/kube-rbac-proxy/0.log"
Nov 01 22:51:18.577402 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.577378    5845 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"]
Nov 01 22:51:18.580546 ip-10-0-135-146 kubenswrapper[5845]: W1101 22:51:18.580520    5845 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c6e83f1_3ddd_443a_887f_606c14dd5c72.slice/crio-3c1dd9329e1fdf7f549e9461441507f741696607f8b5f987c02306ea35139579 WatchSource:0}: Error finding container 3c1dd9329e1fdf7f549e9461441507f741696607f8b5f987c02306ea35139579: Status 404 returned error can't find the container with id 3c1dd9329e1fdf7f549e9461441507f741696607f8b5f987c02306ea35139579
Nov 01 22:51:18.583581 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.583554    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/kube-rbac-proxy-thanos/0.log"
Nov 01 22:51:18.624175 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.624149    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6db8848b-ac26-44f3-855f-a5217ee57d81/init-config-reloader/0.log"
Nov 01 22:51:18.832236 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.832159    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/thanos-query/0.log"
Nov 01 22:51:18.854075 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.854045    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/kube-rbac-proxy-web/0.log"
Nov 01 22:51:18.877921 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.877897    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/kube-rbac-proxy/0.log"
Nov 01 22:51:18.901078 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.901052    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/prom-label-proxy/0.log"
Nov 01 22:51:18.923901 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.923873    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/kube-rbac-proxy-rules/0.log"
Nov 01 22:51:18.947278 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:18.947261    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fb8c8bd9c-6mv4x_f109cd89-d922-4952-b092-0360bbbbaf33/kube-rbac-proxy-metrics/0.log"
Nov 01 22:51:19.281586 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:19.281555    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm" event={"ID":"1c6e83f1-3ddd-443a-887f-606c14dd5c72","Type":"ContainerStarted","Data":"31bd7fe9f6fa45f68e3e5eb1d45c82e010199a257cd8c190fd65c4e5dc5c9842"}
Nov 01 22:51:19.281751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:19.281602    5845 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm" event={"ID":"1c6e83f1-3ddd-443a-887f-606c14dd5c72","Type":"ContainerStarted","Data":"3c1dd9329e1fdf7f549e9461441507f741696607f8b5f987c02306ea35139579"}
Nov 01 22:51:19.281751 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:19.281697    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:19.301008 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:19.300962    5845 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm" podStartSLOduration=2.300951016 podStartE2EDuration="2.300951016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:19.299728452 +0000 UTC m=+3982.555908232" watchObservedRunningTime="2026-11-01 22:51:19.300951016 +0000 UTC m=+3982.557130779"
Nov 01 22:51:23.181476 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:23.181436    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jp45_55adb8b3-868b-4fe9-a2ee-a2de94739289/dns/0.log"
Nov 01 22:51:23.201245 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:23.201185    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jp45_55adb8b3-868b-4fe9-a2ee-a2de94739289/kube-rbac-proxy/0.log"
Nov 01 22:51:23.334837 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:23.334815    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r86vn_baf3bf5f-cbda-43b8-b263-dad640f9f5a3/dns-node-resolver/0.log"
Nov 01 22:51:24.103932 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:24.103903    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7hr8_8d602079-f8c5-49f7-afe1-8d6e074b8c27/node-ca/0.log"
Nov 01 22:51:25.297371 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:25.297344    5845 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gx6ks/perf-node-gather-daemonset-tkzkm"
Nov 01 22:51:28.144907 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:28.144870    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-kdssh_ed7e30f6-adec-4f93-a098-2ab4f61f937d/kube-auth-proxy/0.log"
Nov 01 22:51:28.278905 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:28.278782    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fp2b7_72fb2f07-f693-45bc-90e5-f622fdef72e5/istio-proxy/0.log"
Nov 01 22:51:28.420798 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:28.420754    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-mrlcb_73a30e41-ed57-4bff-92a2-38160219bfb5/istio-proxy/0.log"
Nov 01 22:51:28.447668 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:28.447626    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-prd6f_b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec/istio-proxy/0.log"
Nov 01 22:51:31.411339 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:31.411287    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b69dc6f4d-kdssh_ed7e30f6-adec-4f93-a098-2ab4f61f937d/kube-auth-proxy/0.log"
Nov 01 22:51:31.466448 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:31.466418    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-fp2b7_72fb2f07-f693-45bc-90e5-f622fdef72e5/istio-proxy/0.log"
Nov 01 22:51:31.537653 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:31.537626    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-mrlcb_73a30e41-ed57-4bff-92a2-38160219bfb5/istio-proxy/0.log"
Nov 01 22:51:31.555758 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:31.555728    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-prd6f_b81dfdb2-ef94-4de7-9b0a-b8de7d5a09ec/istio-proxy/0.log"
Nov 01 22:51:31.678119 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:31.678048    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sn9hx_a2ddd68d-9250-4231-8b29-ee881e0faeae/serve-healthcheck-canary/0.log"
Nov 01 22:51:32.369421 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:32.369386    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgzv8_e7d58a7a-ace5-47e8-9d39-6ffcf04804af/kube-rbac-proxy/0.log"
Nov 01 22:51:32.391853 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:32.391827    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgzv8_e7d58a7a-ace5-47e8-9d39-6ffcf04804af/exporter/0.log"
Nov 01 22:51:32.415001 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:32.414981    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bgzv8_e7d58a7a-ace5-47e8-9d39-6ffcf04804af/extractor/0.log"
Nov 01 22:51:34.822980 ip-10-0-135-146 kubenswrapper[5845]: E1101 22:51:34.822938    5845 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bgzv8" containerName="extractor"
Nov 01 22:51:35.134033 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:35.133949    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-jd65m_60df8928-8bda-496d-a064-429629ed2583/manager/0.log"
Nov 01 22:51:35.156734 ip-10-0-135-146 kubenswrapper[5845]: I1101 22:51:35.156708    5845 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-98c48f84c-jd65m_60df8928-8bda-496d-a064-429629ed2583/copy-manifests/0.log"